How we set up the culture around this, how we are able to scale this fast and how will of this help you in improving your testing program and it’s culture.

1. Testing culture: making testing look easy!?
Martijn Scheybeler / Lead Growth, SEO & Analytics / martijn@thenextweb.com / @MartijnSch

2. EASY, WHAT?

3. • Leads Marketing: Growth, SEO & Analytics
• Marketing team The Next Web: 12 (60+)
• Into testing for the last 3 years
• Weekly running 10 new tests, we can do 15-20 :(
• Scaling, coding and automating
ConversionSUMMIT 2015 - @MartijnSch
Who is this guy?

4. What is The Next Web?
ConversionSUMMIT 2015 - @MartijnSch

5. 7 million unique users, 13 million pageviews

6. TNW Conference: Amsterdam & New York: 20.000 attendees

7. TNW Deals: The online marketplace for digital tastemakers

8. TNW Academy: Hand-picked online courses to improve your professional life

9. ConversionSUMMIT 2015 - @MartijnSch
• From no testing to …? in 6 months
• Setting up a testing culture from scratch
• Scaled from: 1 test a month > 10 a week
• 10 NEW tests as week, the ability to do: 15
What qualifies me to speak here?

10. ConversionSUMMIT 2015 - @MartijnSch
• How to create a culture of testing?
• How were / are we able to scale so fast?
• How it influences our (company / team) culture?
• Summary: How do YOU scale your testing program?
What do I like to talk about…

11. ConversionSUMMIT 2015 - @MartijnSch
• Team structure & dynamics
• Creating experiment ideas + planning
• Creating & coding experiments
• Analysing experiments
• What’s next?
… and it goes something like this

12. ConversionSUMMIT 2015 - @MartijnSch
• Design, Development, QA, Project Management, Legal
> CEO approval: you’re doing it wrong!
• Can you design, code & analyse? You don’t need a team.
• Cross-functional or not?
Team structure

13. ConversionSUMMIT 2015 - @MartijnSch
TNW Style: Team structure

14. ConversionSUMMIT 2015 - @MartijnSch
• 1 Web Analyst: Designing, Coding & Analysis
• 2 Interns: Analysis + Coding
• Me: Strategy, Coding, Analysis
• Support: 1 designer + 1 front-end developer
Not cross functional (at the moment)
TNW Style: Team structure

15. ConversionSUMMIT 2015 - @MartijnSch
• “Our developers won’t let us change the code”
• “I need to run our tests against the legal and *BS* team”
• “We’re still waiting for the design reviews”
• “We don’t think A/B testing can improve our results”
• “I asked my girlfriend to test the site, she didn’t have issues”
Team dynamics

16. What does our culture looks like?

18. ConversionSUMMIT 2015 - @MartijnSch
• “This is how you access our live servers, need GTM as well?”
• “Legal, who?”
• “We still have 5 design reviews laying around, DAMMIT!?”
• “10 new tests this week! YOLO!”
TNW Style: Team dynamics

19. ConversionSUMMIT 2015 - @MartijnSch
• A lot of ideas, but no REAL ideas: button colours?
• =AVG(TIME*0,272+EFFORT*0.123+HAIRCOLOUR*0,605)
• A planning heavily discussed with the …
Experiment ideas + planning

20. Make sure you don’t have a HIPPO in the room

21. Boris (CEO) “I had a very good idea this weekend! Can we make experiments more aggressive.”

22. ConversionSUMMIT 2015 - @MartijnSch
• Ok, we do have a planning
• “It doesn’t need to be rocket science”
• We don’t have a huge backlog
TNW Style: Planning, Planwhat?

23. ConversionSUMMIT 2015 - @MartijnSch
• Design by committee, according to style guides
• Won’t get you a lot further
• Outsourced testing? In-house teams (with no time)?
Creating & coding experiments

24. ConversionSUMMIT 2015 - @MartijnSch
• Designer: “I created a new design, we should test it!”
• Developer: “I looked at your A/B testing script: refactored it”
• Analytics: “We have a backlog but we could use new ideas”
• Me: “Can we do 20 tests a week?”
TNW Style: Creating experiments

25. Deploying in 3..2..1! Flexibility v.s. Scale v.s. €0 Costs

26. ConversionSUMMIT 2015 - @MartijnSch
• Designed by our CRO team
• Working with our design team to test very big things
• Responsible: no way of hiding behind bad code
TNW Style: Creating experiments

27. ConversionSUMMIT 2015 - @MartijnSch
• Google Tag Manager
• A/B Testing & Multivariate Testing: http://tnw.to/testingJS
• Custom JavaScript / jQuery for changes
• Supported by the dataLayer for targeting / segmenting
• You could do better: back-end testing!
TNW Style: Coding experiments

28. ConversionSUMMIT 2015 - @MartijnSch
• Google Analytics? Adobe Analytics?
• You do make sure to check for power + significance, right?
• You’re also tracking secondary KPIs?
• You decide to ‘re-test’?
Analysing experiments

29. Automated reporting via TNW Analytics: -20%

30. ConversionSUMMIT 2015 - @MartijnSch
• Google Analytics API
• Custom Dimensions / Event Tracking
• Automated reporting
• Ultimate pleasure and fun + no annoyed team members
TNW Style: Analysing experiments

31. TNW Analytics: Adding experiments + ideasAutomated reporting via TNW Analytics: -20%

32. ConversionSUMMIT 2015 - @MartijnSch
• Take it easy… do you know what to do?
• First: make friends!
• Second: make sure you know what you’re doing!
• Third: map out your strategy, do you have the resources?
• Worst case? Read back the slides!
What’s next for you?

33. ConversionSUMMIT 2015 - @MartijnSch
• Grow our team: 15 people?
• Cross functional, a mistake?
• Automate & Innovate even more
• Drink more …
TNW Style: What’s next?

34. ConversionSUMMIT 2015 - @MartijnSch
• Create / Embrace a culture of testing: don’t create silos
• Make the whole team + organisation come up with ideas
• Design ideas yourself or ask your designer(s)
• Learn to code so you can create experiments yourself
• Want to scale? Make sure you have the resources to automate
Take aways

35. ConversionSUMMIT 2015 - @MartijnSch
Martijn Scheybeler
Lead Growth, SEO & Analytics - TNW
Email: martijn@thenextweb.com
Twitter: @MartijnSch
Have fun! Thank you!
Slides: http://tnw.to/csummit
Code: http://tnw.to/testingJS

36. ConversionSUMMIT 2015 - @MartijnSch