Digital Data Tips Tuesday
•Download as PPTX, PDF•
0 likes•809 views
Our testing culture & our testing program: How we created a culture of testing? How were / are we able to scale so fast? How it influences our (company / teams) culture? Summary: How can YOU scale your testing program?
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (7)
Similar to Digital Data Tips Tuesday
Similar to Digital Data Tips Tuesday (20)
Recently uploaded
Recently uploaded (20)
Digital Data Tips Tuesday
- 1. Our testing culture & our testing program Martijn Scheybeler / Leads Growth, SEO & Analytics / martijn@thenextweb.com / @MartijnSch
- 2. • How we created a culture of testing? • How were / are we able to scale so fast? • How it influences our (company / teams) culture? • Summary: How can YOU scale your testing program? What do I like to talk about… Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 3. • Why we do what we do? • Team structure & dynamics • Creating experiment ideas + planning • Creating & coding experiments • Analysing experiments • What’s next? … and it goes something like this Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 4. Why we do what we do? Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 5. It’s all about: Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 6. VELOCITY! Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 7. or in other words: Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 8. FAIL OFTEN, FAIL FAST Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 9. Team Structure & Dynamics Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 10. • Design, Dev, QA, Project Management, Legal, CEO? YOU’RE DOING IT WRONG! • Can you design, code & analyse? You don’t need a team. • Cross-functional or not? Team structure Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 11. Team structure Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 12. • 1 Web Analyst: Designing, Coding & Analysis • 2 Interns: Coding, Analysis • Me: Strategy, Coding, Analysis • Support: 1 designer + 1 front-end developer. Not cross functional (at the moment)! Team structure Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 13. What does our culture looks like?
- 14. • Better ask for forgiveness, than permission. • F*CK IT, WE’LL DO IT LIVE! • It’s not rocket science! • We’ll do a much as possible inhouse + open. TNW ‘Core Values’ Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 15. • “This is how you access our live servers, need GTM as well?” • “Legal, who?” • “We still have 5 design reviews laying around, DAMMIT!?” • “10 new tests this week! YOLO!” Team dynamics Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 16. “You don’t have decades of experience I see. This is not how it works in the real world.” Digital Data Tips Tuesday - #ddtt - @MartijnSch - Random Enterprise Employee
- 18. Take away: make sure you have buy in!
- 19. Boris (CEO) “I had a very good idea this weekend! Can we make experiments more aggressive.”
- 20. Experiment Ideas + Planning Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 21. “ We don’t want you to run this test! ” Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 22. • Ok, we do have a planning • “It doesn’t need to be rocket science” • We don’t have a huge backlog Ideas + Planning, Planwhat? Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 23. • Designer: “I created a new design, we should test it!” • Developer: “I looked at your A/B testing script: refactored it” • Analytics: “We have a backlog but we could use new ideas” • Me: “Can we do 20 tests a week?” Creating experiments Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 24. Deploying in 3..2..1! Flexibility v.s. Scale v.s. €0 Costs
- 25. • Designed by our CRO team • Working with our design team to test very big things • Responsible: no way to hide behind bad code • Responsible: communicate everything to the company Creating experiments Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 26. </Coding Experiments> Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 27. • Google Tag Manager • A/B Testing & Multivariate Testing: http://tnw.to/testingJS • Custom JavaScript / jQuery for changes • Supported by the dataLayer for targeting / segmenting • We could do better: back-end testing! Coding experiments Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 28. Analysing Experiments Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 29. Automated reporting via TNW Analytics: -20%
- 30. eas, Backlog, Documentation, Analy Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 31. • Send variant through Custom Dimensions • Track everything with event tracking • ‘Tracking plans’ so we also know what we track Google Analytics Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 32. • Google Analytics Reporting API • Custom Dimensions / Event Tracking / Goals • Automated reporting • Ultimate pleasure and fun + no annoyed team members Analysing experiments Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 33. TNW Analytics: Adding experiments + ideasAutomated reporting via TNW Analytics: -20%
- 34. • Grow our team: 20 people? • Cross functional, get developers on board, a mistake? • Automate & Innovate even more • Drink more …? What’s next? Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 35. Create / Embrace a culture of testing: don’t create silos Make the whole team + organisation come up with ideas Learn to code so you can create experiments yourself / faster Want to scale? Make sure you have the resources to automate Take aways Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 37. Martijn Scheybeler Lead Growth, SEO & Analytics - TNW Email: martijn@thenextweb.com Twitter: @MartijnSch Thank you! Slides: http://slideshare.net/MartijnScheijbeler Code: http://github.com/martijnsch/cro Digital Data Tips Tuesday - #ddtt - @MartijnSch
- 38. Digital Data Tips Tuesday - #ddtt - @MartijnSch
Editor's Notes
- Thanks a lot for having me here, I’m excited to give you some insights into how to build and/or improve the testing culture within your organisation to make sure it looks easy for you to keep on testing.
- What I’d like to talk about is how we set up the culture around this, how we are able to scale this fast and how will of this help you in improving your testing program and it’s culture.
- So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
- So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
- So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
- So how does your team structure look like? What I see a lot is that it contains so many silos. 1 Project manager to make absolutely sure that office politics is handled. He or She comes up with an idea and runs it by legal, yes this happens, I bet some people here are also to scared to admit it ;-). They ask designers to create a prototype, preferably have 2 rounds of changes. Have it coded by their development team and have their QA person check the site. Total time to run through 1 test: probably around 3 weeks. But who can design, code and analyse? So you can save yourself some time? Let’s do a show of hands on that!
- An overview of how a part of our TNW structure looks like. Design and Development are sitting next to each other and only marketing is sitting on different islands. Mostly because we just have too many people by now.
- Our core team working on CRO consist of 4 people. They’re supported by 1 designer and 1 front-end developer who jump in whenever needed when we want to do stuff that goes above our heads in terms of knowledge.
- Meet our culture, oh and these are our founders/ CEO + COO Boris and Patrick.
- So in our case we turned around these ‘quotes’ as we didn’t have any of them. I don’t even know what the name of our laywer is. We have access right to our live servers to push winning variants live immediately. And in case of design reviews, it’s more that we can’t keep up with coding them then that we’re waiting for them, luckily!
- So in our case we turned around these ‘quotes’ as we didn’t have any of them. I don’t even know what the name of our laywer is. We have access right to our live servers to push winning variants live immediately. And in case of design reviews, it’s more that we can’t keep up with coding them then that we’re waiting for them, luckily!
- So how does your team structure look like? What I see a lot is that it contains so many silos. 1 Project manager to make absolutely sure that office politics is handled. He or She comes up with an idea and runs it by legal, yes this happens, I bet some people here are also to scared to admit it ;-). They ask designers to create a prototype, preferably have 2 rounds of changes. Have it coded by their development team and have their QA person check the site. Total time to run through 1 test: probably around 3 weeks. But who can design, code and analyse? So you can save yourself some time? Let’s do a show of hands on that!
- Every presentation needs a quote by Tony Robbins. We highly believe that all the excuses you make belong to this problem. You can always find a solution for your problems whatever the issue is.
- HIPPO, the Highest Paid Person in the Organisation. Worst case I saw is that in calculating the averages the HIPPO’s opinion had more weight. You gotta be kidding me!
- Our HIPPO works in a different way, meet Boris again, our CEO. He obviously likes to share his opinion as well but does that in a different way. It happens at least monthly that he walks up to me with another crazy idea where we completely change the whole structure again. I rather have this over him prioritising our tests ;-)
- So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
- So how does your team structure look like? What I see a lot is that it contains so many silos. 1 Project manager to make absolutely sure that office politics is handled. He or She comes up with an idea and runs it by legal, yes this happens, I bet some people here are also to scared to admit it ;-). They ask designers to create a prototype, preferably have 2 rounds of changes. Have it coded by their development team and have their QA person check the site. Total time to run through 1 test: probably around 3 weeks. But who can design, code and analyse? So you can save yourself some time? Let’s do a show of hands on that!
- So I have to admit we don’t have that really structured planning, we might want to have it. But as we can test on a quite fast pace it can be that all the new ideas that we come up with can be tested within a month. Which means at this point we prefer not to make our prioritisation process to hard on ourselves.
- That’s where our ‘support troops’ come in handy. Our developers and designers can assist us wherever possible in finding us new improvements in what we could improve and also make sure it’s coded.
- But what if you could have an almost unlimited number of tests running at the same time against at almost 0 costs.
- That’s why creating experiments is easy, we can code and design them ourselves and only if it’s a really big change we try to involve our design and development team as well. Which makes it clear that we want good code, to make sure we can’t be blamed for hickups on the site. Which would also shoot us in the foot.
- So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
- Our technical setup makes this earlier quote very easy, all testing for now is ran in Google Tag Manager and supports both A/B testing and MVT testing. So we write the custom jQuery for creating our changes which any tool would do for us anyway. Want to do segmentation on a certain user segment? Sure, probably our dataLayer is already supporting this and sending the right variables for this to GTM. Where we like to improve? We want to be able to do this all from our backend, why? We want to have less flickering, more caching which improves speed and less JS that’s needed to be loaded.
- So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
- We decided to built a small platform where we could one 1 side keep a backlog of tests, move them to real testing documentation if time would come and also keep the data on metrics there. So we only needed 1 tool to keep track of the test and it’s documentation. Cutting out 2 other tools. And this is how it looks like as it retrieved the data for a ‘dummy’ experiment.
- So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
- It’s build on top of the Google Analytics API. As we send all our data for variants to custom dimensions in Google Analytics and data on all our KPIs with event tracking we have all the metrics in there already. Which makes the tool perfect to do automated reporting for us. Creating ultimate pleasure for the team as they don’t have to pull endless reports with the data and then start analysing it.
- It’s build on top of the Google Analytics API. As we send all our data for variants to custom dimensions in Google Analytics and data on all our KPIs with event tracking we have all the metrics in there already. Which makes the tool perfect to do automated reporting for us. Creating ultimate pleasure for the team as they don’t have to pull endless reports with the data and then start analysing it.
- Adding a new idea for a test is easily, we keep track of the dates of the test and the templates/ user segments (or swim lanes, thanks to Andy Nelson of Moz I learned that term) the test is running on. We choose an objective metric and if needed provide more details on the right side of the screen.
- What’s next? We’re currently already growing the marketing team, which means we’ll get more traffic to test on, which means more testing. We might embrace cross functional a bit more. Currently we’re already working more with our developers and designers to make them part of the process and use their skills to fasten up the process. We might even get them on the marketing team later on to make ourselves cross-functional. For the analysis part we like to automate even more and there’s still a lot of improvement.
- So the take aways: creating a great culture where people are taking full responsibility and not blaming silos. Have your whole team work on the process and make sure you completely understand the full process from design to analysis and back. Running more than X tests a week or month. See what you could analyse, otherwise you might get stuck in repetitive work and burn yourself out in the long run.
- Thanks a lot for you time and have fun the rest of the day. Slides are available at this URL, or come talk to me then I’ll make sure you’ll get a link in your inbox. Thanks!