Conversion Jam 2016 - Stockholm

1. How to run 200 tests a year & what do you learn?
Martijn Scheijbeler / Director of Marketing / martijn@thenextweb.com / @MartijnSch

2. Emerce Conversion - @MartijnSch
Title

3. Know why, you test and how
you scale what matters:
@MartijnSch
WHAT YOU GET OUT OF THIS?
We’ll be sharing what tools + process we have at The Next Web to get started!

4. How it all
Started
@MartijnSch
THE START

5. Why we feel
The need for testing?
@MartijnSch#ConversionJam

6. “So what would we have
to do to grow x10?”
Boris (CEO)
“I have no idea what I’m doing”
@MartijnSch / #ConversionJam

7. MEDIA
News
E-commerce
EVENTS
Events
Conferences
WORK SPACES
TQ
INTELLIGENCE
Index

8. Why am I
showing you this slide?
@MartijnSch#ConversionJam

9. But how did we get to
200 A/B Tests a Year?
@MartijnSch#ConversionJam

10. Testing Figures
Data from 2015
2,5CRO Team in 2015
38.000.000Sessions with Tests
0CRO Team in 2014
5-7Tests per week
PricelessAmounts of fun
35%Traffic v.s. Tests
@MartijnSch

11. @MartijnSch
Our roadmap for getting to 200 tests a year
Jan
‘15
Let’s do more testing!
After running 20-30 tests in 2014 we knew we had to scale up.
Resources were low, but with the addition of a web analist we could grow.
Apr
‘15
Hitting some limits
Google Analytics Content Experiments (JS) was our first tool to
use. But soon we hit the limits of the tool.
May
‘15
Setting the vision
After running 40 tests by May we needed a beter process that
could scale and that aligned with our testing vision.
Dec
‘15
Reaching XXX tests
With the testing vision and updated testing program we were able
to run XXX tests in total in 2015.
3 Months
1 Month
7 Months

12. Quantity v.s. Quality?
@MartijnSch
But why would you like to run at high velocity? Why not go for quality?
#ConversionJam

13. Testing Figures
Data from 2015
26%Winning Tests in 2015
@MartijnSch
:(:)

14. “If you double the number of
experiments you’re going to
double your inventiveness”
- Jeff Bezos
@MartijnSch

15. What makes it different:
Publishers v.s. E-commerce?
@MartijnSch#ConversionJam

16. The opportunity is 350-400 tests a
year. As ROI per winning test is
lower we need to run a lot to have
a positive return.
Efficiency <> Costs
@MartijnSch / #ConversionJam

17. How is our (marketing) team
Structured?
@MartijnSch#ConversionJam

18. SEO
CRO/ANALYTICS
TNW DEALS
INDEX.CO
CONTENT
GROWTH
@MartijnSch
Front-end Developers

19. F*CK it, we’ll do it live
It’s not rocket science
“Innovate through in-house solutions”
@MartijnSch

20. What’s our
Process
@MartijnSch
PROCESS

21. We failed,
not once but also not twice…
@MartijnSch#ConversionJam

22. We needed to learn,
Knowledge building?
@MartijnSch
You got to love working with people who are like minded.
#ConversionJam

23. Thanks for all your support!
We learn from you!
@MartijnSch / #ConversionJam

24. The more we test the more
we innovate and the higher
inventiveness.
Find the
perfect process
@MartijnSch / #ConversionJam

25. The opportunity is 350-400 tests a
year. As ROI per winning test is
lower we need to run a lot to have
a positive return.
Know your metrics?
@MartijnSch / #ConversionJam

26. @MartijnSch
1
2
Unique Users
Growing our unique user base will
increase the potential to reach Generation T.
2
3 3
Our metrics/KPIs for success;
User Engagement: PV/Session
A higher engagement with our user base will
increase the number of pageviews per session.
User Retention: Returning %
Increasing the percentage or returning users will
increase our brand awareness.
Newsletter Signups
To get a relationship with our users we’d like to
have their email address for the newsletter.
Social Shares
More shares will influence our reach amongst
new and returning users.

27. The opportunity is 350-400 tests a
year. As ROI per winning test is
lower we need to run a lot to have
a positive return.
Do you have a
swimming pool?
@MartijnSch / #ConversionJam

28. The opportunity is 350-400 tests a
year. As ROI per winning test is
lower we need to run a lot to have
a positive return.
Segment Users
@MartijnSch / #ConversionJam

29. Users are the same
@MartijnSch
KNOW YOUR SEGMENTS
X
X
X

30. Users are the same
@MartijnSch
KNOW YOUR SEGMENTS
X
X
X

31. Google Tag Manager (GTM) is
making our lives easier +
scalable.
What tools we use
@MartijnSch / #ConversionJam

32. So we need to improve most of
our communication around
testing.
Communicate more
@MartijnSch / #ConversionJam

33. How did we
overcome our fails?
@MartijnSch
You fail once, you fail twice!
#ConversionJam

34. We didn’t move to any ‘real’
CRO tool but decided to
iterate on our own: GTM <> JS.
GTM
Download:
Github.com/TheNextWeb/CRO
@MartijnSch / #ConversionJam

35. Building our own solutions for
saving testing documentation
+ analysis of tests.
Innovate
@MartijnSch / #ConversionJam

36. We moved most of our
communication towards
Slack. Making the teams more
involved in the whole process.
Always
communicate
@MartijnSch / #ConversionJam

37. We hired two front-end
developers, to make sure in
the long-run we can test
bigger things.
Front-end Dev
@MartijnSch / #ConversionJam

38. 5-7Tests per week
2-4Tests per week
30-40Tests per year
20%More efficient
@MartijnSch
Improvement
What does progress look like?

39. What’s in our
Future?
@MartijnSch#ConversionJam

40. Who’s able to deploy winning
variants within minutes from their
codebase?
Continous
Improvement
@MartijnSch / #ConversionJam

41. This will make segmentation +
automation even easier. In addition
it will decrease the flickering effect.
Back-end
Testing
@MartijnSch / #ConversionJam

42. Testing small doesn’t work, so we
need to grow even bigger. On
average we grow traffic with around
10-20% yearly. At the same rate we
can increase our testing program.
Big > Biggest
@MartijnSch / #ConversionJam

43. TL;DR
Process
1. Have the right company vision
2. Have the right team structure
3. Have access + support from C level
4. Create partnerships with like minded
5. You’ll have trouble as well, nobody’s perfect
6. Don’t be afraid to invent the wheel again
7. Always focus on innovation
@MartijnSch#ConversionJam

44. What tests
did we learn from
@MartijnSch
LEARNINGS

45. MEDIA
News
E-commerce
EVENTS
Events
Conferences
WORK SPACES
TQ
INTELLIGENCE
Index

46. Sharing
@MartijnSch#ConversionJam
More content on social media drives
brand awareness + reach

47. Sharing brings in more loyalty,
more engagement and
hopefully more traffic.
Share buttons

48. • Variant increase clicks with 16%
• Had a significance + power level of > 99%
• We ran the test for 1x 7+ days
• Targetting over 200.000 users
Winner: Variant
Original:
Variant

49. Engagement
@MartijnSch#ConversionJam
More engagement increase loyalty
which provides advertising revenue

50. Engagement is important,
focus on getting more clicks
on the related stories in the
sidebar.
Rankings!

51. • Variant increase clicks with 36%
• Had a significance + power level of > 99%
• We ran the test for 7+ days
• Targetting over 300.000 users
Winner: Variant 1
Original: Variant:

52. Engagement is important,
focus on getting more clicks
on the related stories below
the article.
Related Stories

53. • Variant increase clicks with 36%
• Had a significance + power level of > 99%
• We ran the test for 2x 14+ days
• Targetting over 500.000 users
Winner: Variant
Original:Variant

54. So what if we remove the
whole sidebar, wouldn’t people
be more focused on the stuff
that is out there?
Big, bigger

55. • Variant decreased clicks with 17% + 14%
• Had a significance + power level of > 99%
• We ran the test for 14+ days
• Targetting over 420.000 users
Winner: Original

56. Now you know how to run
200 A/B Tests a Year?
@MartijnSch

57. But did we get to
200 tests?
@MartijnSch

58. “Why didn’t we run 200 tests?”
“What went wrong?”
“Do I need to start crying?”
Boris (CEO)
@MartijnSch / #ConversionJam

59. “Well, if this guy wouldn’t have stuck
his fingers into a lawn mower, YES!”
Missing out on ±20 tests in 2015.
The Lawn Mower
@MartijnSch / #ConversionJam

60. Take Aways:
1. Focus on running X tests a year
2. Improve the process
3. Have the right (free) tools
4. Have the right people + support
5. Document/Communicate everything
6. Know what the future looks like
7. Have fun!
@MartijnSch#ConversionJam

61. Emerce Conversion - @MartijnSch
Title

62. Tack!
Ha så kul!
@MartijnSch#ConversionJam

63. Do you have any
Questions?
@MartijnSch#ConversionJam