Growth = ExperimentationI’m going to make you better at testing – whether you are at a big company or a small, I’ve got some information about testing that can help.Testing & Experimentation is important – we take it very seriouslyHere’s a few things that can help
Your product is made up of a bunch of interactions with real people who value the service you provide. Your product is data. The more of this data you have, the faster you achieve confidence on tests. ** Different companies in the audience will be of different sizes, but for Shutterstock in a highly commoditized industry, data is everything.
1TB per day3 downloads a second50
Edimology of smoke tests – you plug in rado
Why does this matter? Sometimes it’s fun to put things on a scale to show differences in opinion. Of course none of these brands would appreciate me doing this, but let’s check it out to prove a point. Problem to solve – Unclear lines between marketing and productMarketing design not yet good at conversionMarketing assets look different than websiteHow do we create a testing culture at Shutterstock
Leadership team isn’t involved in tactics of testing.
This comes into effect if a test is strategic. Because you might be testing strategy – than a flat to win is good. If you are running a revenue test, than only win is what we go on. Because there is a cost to switching to a new design.
1-2mSmall changes matter. Spot the difference, it’s worth 2million dollars.
Got our ass kicked on this – lost potentially 3m Browse account modal- existing features don’t always work
It’s important to test both iterative changes and big swings in some cases, or you may only make small changes.
There’s an “ideas” list sitting on on our internal wiki with 200+ test ideas waiting to go.
Define Highest paid person opinion
Don’t’ overbuild or conduct full redesigns when all you need are optimizations. It’s too much work to take a swing like that.
Here’s a slide from Joshua Porter – really bright guy Why do I drive this point home – because I’ve seen testing cultures where the teams literally can’t launch a new feature. They are so entrenched in the mindset of optimization that a new product is out of the question.
Seriously Advanced A/B Testing by Wyatt Jenkins
Is simply a series of experiments
Wyatt Jenkins – VP Product
How Shutterstock Tests Everything
and Stays Nimble
VP, Product Development
Computer science, Interaction Design, Product Strategy
M.I.T. & University of Colorado
Partner / Board Member of Beatport