Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Is This Normal? Common Challenges Testing Programs Face


Published on

Ryan Garner, Co-Founder of Clearhead
Jessica Vasbinder, Senior Director at Warner Music Group

You are doing it by the book. After a thoughtful search, you choose and implement Optimizely. You've built organizational buy--in for testing, and your colleagues are now sending you their hypotheses. You have even run your first set of tests and perhaps found a winner. Your business is growing and you are data driven now... right?

For many organizations, it's not that easy. The establishment of needle moving optimization programs inside of often complex organizations with federated departments who all feel at least partial ownership of testing is challenging and full of growing pains and existential crises along the way.

In this discussion, we get real and offer practical perspective on the common and healthy optimization program challenges many organizations face. Example scenarios include:

1) We've run a bunch of quick, easy tests and nothing is really moving the needle. Where's the lift?
2) We have more ideas than we could ever possibly test. How do we prioritize?
3) We launched a test that broke stuff on our site! I thought building tests was easy?
4) We said that we should test everything but now realize that is impossible. Does anyone really test everything? Is this program working? How do we keep score?
5) We've had a few winners and now our bosses want to know the annual growth we can expect from them. How do we answer that?

Ryan Garner, Co--Founder of Clearhead, and Jessica Vasbinder, Senior Director at Warner Music Group, discuss what healthy and winning optimization programs really look like based on the experiences of great brands like Warner Music Group, FreshDirect, Blu Dot and more.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Is This Normal? Common Challenges Testing Programs Face

  1. 1. Is This Normal?
 Debunking myths & getting real about the challenges optimization programs face.
 Ryan Garner for Clearhead Jessica Vasbinder for Warner Music Group
  2. 2. Hi. I’m Ryan Garner.! This is Jessica Vasbinder. ! ! This is the part where we tell you about ourselves. ! ! ! (@ryantgarner) (@entweetment)
  3. 3. You’ve seen these examples, right?
  4. 4. +125% lift! control variation
  5. 5. +5,500% lift! control variation
  6. 6. And we’ve all played
  7. 7. Jocelyn Michael
  8. 8. Winner! Good for you! Jocelyn Michael
  9. 9. Now pick which is
  10. 10. A. B.
  11. 11. Everything is awesome!
  12. 12. This stuff is easy,
  13. 13. Let’s get real.
  14. 14. And we are going to do this by asking 1 simple question along the way... ! ! ! ! !
  15. 15. Is this normal?
  16. 16. We ran our first set of tests. Made buttons bigger, changed copy, tested different colors. And zip, zero… ! Is this normal? ! ! !
  17. 17. Yes!
  18. 18. • It’s okay to crawl before you walk, but set expectations. – You may (probably will) NOT materially improve your bottom line. ! • Risk (Disruption) & Reward (Lift) are correlated. – Small tests often provide small returns…especially for smaller businesses!
  19. 19. We don’t know where to start testing or which ideas are good and which are bad or how to keep momentum. ! Is this normal? ! ! !
  20. 20. Very!
  21. 21. You have no doubt heard..
  22. 22. • “Always start at the bottom of the funnel.” • “Test the whole funnel simultaneously.” • “NO, Find the bottlenecks IN the funnel.” • “Navigation is always a good place to test.” • “Forms, baby, forms.”
  23. 23. We say...start with the best ideas!
  24. 24. And where do the best ideas come from?
  25. 25. Look Inward Look Outward Your product roadmap User tests! Your customers’ feedback Competitors Ideas from around the company Vendors trying to sell you stuff you Your previous test results Companies you admire
  26. 26. OMFG! We have 1,000+ testing ideas. Lots and lots of them seem good. Now where do we start? ! Is this normal? ! ! !
  27. 27. Normal? No. Awesome? Hell Yes!
  28. 28. • Create a simple scoring system and get the cream to the top. – It should need no real explanation. – No new acronyms! – Why? No hurt feelings & better ideas!! ! • Then use your own judgment. ! • Repeat regularly.
  29. 29. We launched a test that broke our site! I thought building tests was easy. ! Is this normal? ! ! !
  30. 30. No comment.
  31. 31. • Let’s be real here. When you launch a test, you are pushing new code onto your production website. – Should just anyone be doing this? ! • Do you really want a high velocity, high returning optimization program? – Of course you do! Otherwise it’s just your normal dev process. ! • So mitigate the risks by.. – Using your best developers and designers. Not your marketers –Monitor, Monitor, Monitor.
  32. 32. Leadership wants a growth estimate from all our winning tests, but when we add it up, it seems too good to be true. ! Is this normal? ! ! !
  33. 33. Yes, but…
  34. 34. • There is no perfect way to do this. ! • Stress the difference between statistical significance and observed lift. ! • Trust that there is improvement and move on. – The only accurate way to confirm lift is to re-test. – Do you really want to do that? ! • And if you must show something…
  35. 35. ***
  36. 36. So what did we learn? You are normal if…
  37. 37. • It’s not as easy as you though it was going to be. • Your initial efforts don’t double your business. • You struggle to consistently find great testing ideas. • You break stuff. • You are not sure how to accurately measure your return.
  38. 38. And remember testing is awesome!
  39. 39. Questions?
  40. 40. Thank you. Ryan Garner Jessica Vasbinder