Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Quick Tips for Better Testing - 2016 Bridge Conference

151 views

Published on

You know you need to do more with your digital channels - but what is it and how can you maximize your results? The answer: Test! Attend this session to learn how to maximize results from your testing efforts and discover new ways to make your tests statistically valid, design better tests, and even find value in failure. With real case studies from WWF and other organizations like Special Olympics, Human Rights Watch and the American Diabetes Association, among others, this session will leave you with numerous ideas to test for success. Digital Handout: http://inahat.co/testing-handout

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Quick Tips for Better Testing - 2016 Bridge Conference

  1. 1. Quick Tips for Better Testing
  2. 2. Jen Boland Analytics and Optimization Lead Sara Hoffman Web Development Manager Brian Rogel Associate Director of SEO and SEM About us #Bridge16 Avon Breast Cancer Crusade Beaconfire REDBeaconfire RED
  3. 3. CONFIDENTIAL 3 Agenda • Creating a plan • Test strategy • Test setup • Things to do while your are testing • How to evaluate results • What to do after testing
  4. 4. Create a Plan
  5. 5. Just because someone else did it doesn’t mean it will work for you. CONFIDENTIAL5
  6. 6. Define Website Pain Points 1) Determine goals 2) List problem areas 3) Rank list based on costs and benefits #Bridge16
  7. 7. Review Metrics of the Proposed Test Key Metrics: • Unique Visits • Pageviews • Conversion Rate • Average Gift #Bridge16
  8. 8. Form Analytics & Error Messaging #Bridge16
  9. 9. #Bridge16 FPO
  10. 10. Usability Testing
  11. 11. User Testing Tools #Bridge16
  12. 12. Test Strategy
  13. 13. Determine Metrics that can be Tested #Bridge16 TOP-FUNNEL (Landing Pages) MID-FUNNEL (Transition Pages) BOTTOM-FUNNEL (Conversion Forms) User’sJourney Decision Tips: • Top-funnel tests run faster than bottom-funnel tests • Testing conversions is not always possible • Make sure the target sample size can be reached in the proposed test duration
  14. 14. Determining Sample Size Sample Size Tips: • Use sample size calculators • Run tests until the sample size is reached, not until statistical significance is reached #Bridge16
  15. 15. A minimum detectable effect (MDE) is the smallest true effect that has a ‘good chance’ of being found to be statistically significant.
  16. 16. Test Across Thank You Pages & Emails C: Honor GiftB: Sustainer w/ Premium A:Sustainer
  17. 17. Test Duration: Minimum Minimum Test Duration Tips: • Run tests for full week cycles • One week minimum is possible for websites with exceptionally high traffic • A two week minimum is preferred for most websites #Bridge16
  18. 18. Test Duration: Maximum Maximum Test Duration Tips: • Clearing cookies can cause sample pollution o Lifehacker Poll: 72.35% of people clear cookies or use private browsing o Microsoft Report: studies show 30% to 40% of people clear cookies monthly • Avoid running tests longer than four weeks #Bridge16
  19. 19. Test Duration and Skewed Results #Bridge16 Avoid running tests during holidays or seasonal anomalies (Ex: End-of-year and non-end-of-year traffic acts very differently)
  20. 20. Test Duration and Sample Size Recap #Bridge16 • Determine target sample size • Run tests for full week cycles • 2 to 4 week test durations are best • Run tests until the target sample size is reached, not until statistical significance is reached • Be aware of holidays and seasonal anomalies
  21. 21. #Bridge16 Where are you going? Make a map January February March April May June July Homepage User Testing Homepage Hero Slider A/B Homepage Static Hero Build Hero Ask String High-Low Survey - Legal Survey Rebuild overlay Pre Year-End Overlay POC  possible extension Action TY Story Title Suggested Amount inahat.co/bridge-roadmap
  22. 22. CONFIDENTIAL 25 Test Variation Name Test Length Traffic % # of Visitors VWO Type Test Location ETA for Results Hero Homepage Donation Ask No Test 7 60% 587 ABCD CMS Homepage 1/12/2015 Hero Homepage Donation Ask Control 7 13% 1401 ABCD CMS Homepage 1/12/2015 Hero Homepage Donation Ask Var1 7 13% 1458 ABCD CMS Homepage 1/12/2015 Hero Homepage Donation Ask Var2 7 13% 1464 ABCD CMS Homepage 1/12/2015 Grey Overlay Homepage Real Control 7 50% 883 Split test CMS Homepage 1/20/2015 Grey Overlay Homepage Grey Overlay 7 50% 857 Split test CMS Homepage 1/20/2015 Hamburger Control 14 50% 1828 ABC Homepage and Directory 6/8/2015 Hamburger Button 14 25% 920 ABC Homepage and Directory 6/8/2015 Hamburger Left Logo 14 25% 943 ABC Homepage and Directory 6/8/2015
  23. 23. Create a Hypothesis CONFIDENTIAL26
  24. 24. Hypothesis Creation #Bridge16 Sections of a Hypothesis: 1) Statement of Problem 2) Proposed Solution 3) Expected Results
  25. 25. Hypothesis Template #Bridge16 Based on <data discovered> we believe <website issue> is causing <undesirable result>. We believe <change to be tested> will help resolve <website issue> and cause <desired result>.
  26. 26. Writing your Hypothesis #Bridge16 Example Hypothesis: Based on a 31% below average conversion rate for paid search traffic we believe a lack of information on donation forms is causing a high drop-off rate. We believe sending people to an information rich splash page before the donation form will help resolve the high drop-off rate and cause an increase in conversion rate.
  27. 27. Paid Search Landing Page Test #Bridge16 +47% Conversion rate Less form fields More information
  28. 28. Test Setup CONFIDENTIAL31
  29. 29. Testing Tools #Bridge16
  30. 30. Test Setup • Preserve Query Parameters • Link with Analytics Tools #Bridge16 https://testsite.org?Donation2?d f_id=12362&12362.donation=for m1&utm_source=newsletter&ut m_medium=email&utm_content =link1&utm_campaign=fall
  31. 31. Beware of Peeping Toms • Browser Tools like WAPPALYZER, dataSlayer, Pessimezely or WASP can expose your test and variant names! #Bridge16
  32. 32. Segmentation #Bridge16
  33. 33. Test Large to Small #Bridge16
  34. 34. Test Broadly Test VariationControl #Bridge16
  35. 35. Don’t Work Late or on Weekends #Bridge16
  36. 36. While You’re Testing…
  37. 37. Monitor Results #Bridge16
  38. 38. Monitor Results #Bridge16
  39. 39. Micro Conversions #Bridge16
  40. 40. Micro Conversions #Bridge16 Control Variant +9.7% Website sign-in clicks +4.9% Credit card sign-in clicks +21% Registration clicks
  41. 41. Avoid Other Website Changes #Bridge16
  42. 42. Results
  43. 43. #Bridge16 Tools to Help with Testing – Handout! Calculators!
  44. 44. Tools to Help with Testing – Handout! Your Good Friend – Excel! Percent Change (Current Value – Previous Value) / Previous Value #Bridge16
  45. 45. Unanticipated Results #Bridge16
  46. 46. P-Value and Statistical Power #Bridge16
  47. 47. “If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time.” - David Colquhoun
  48. 48. Analyzing Data: What’s a Win? #Bridge16 Checklist:  Low P-Value  High Power  Test Reaches Target Sample Size  Test is Not an Industry Outlier
  49. 49. What Tools Tell You #Bridge16
  50. 50. Two-Tail T-Test
  51. 51. After Your Test
  52. 52. Validate – Iterate! inahat.co/bridge-validate Hypothesis Create / Iterate Continuous Testing Measure Results #Bridge16
  53. 53. #Bridge16 Modify Your Map January February March April May June July Homepage User Testing Homepage Hero Slider A/B Homepage Static Hero Build Hero Ask String High-Low Paid Search Ask DRTV Ask Survey - Legal Survey Rebuild overlay Pre Year-End Overlay POC  possible extension Paid Search Overlay Action TY Story Title Suggested Amount inahat.co/bridge-roadmap
  54. 54. Sharing Lessons Across Channels #Bridge16 Using Google AdWords to test how effective different copy variations are at improving click-through-rates. Control Variants +25.2% CTR Improvement -6.9% CTR Reduction
  55. 55. Sharing Lessons Across Channels #Bridge16 The difference in “Don’t miss” and “Don’t miss out on” made a 34.4% difference in click-through-rates. The test was significant at 99% with 99.93% statistical power. Variant +34.4% CTR Improvement Control
  56. 56. Cross-Channel Implementations #Bridge16 Copy tests help determine the best messaging for your audience. Improvements discovered can be leveraged in many channels: Meta Content (Organic) Ad Copy (Paid) • Email • Direct Mail • DRTV • Social Media • Organic Meta Content • Display Ad Copy
  57. 57. I’ve failed over and over and that is why I succeed. CONFIDENTIAL60 Michael Jordan
  58. 58. Questions? Download our Free Testing Guide: http://inahat.co/testing-handout Don’t forget to visit the Solutions Showcase! Many of the ideas discussed today are on display at the Solutions Showcase!

×