How	
  to	
  	
  Increase	
  Your	
  Tes0ng	
  Success	
  by	
  
Combining	
  Qualita0ve	
  and	
  Qualita0ve	
  Data
Hite...
Hiten	
  Shah	
  
Co-­‐Founder	
  
KISSmetrics	
  
@hnhshah	
  
Hiten	
  Shah	
  
Co-­‐Founder	
  
KISSmetrics	
  
@hnhshah	
  
There’s	
  a	
  big	
  problem	
  with	
  A/B	
  tes5ng!
You’ve	
  tested	
  the	
  color	
  of	
  your	
  bu?ons
You’ve	
  watched	
  your	
  compe5tors
You’ve	
  tried	
  tes5ng	
  everyone’s	
  opinion
You’ve	
  hit	
  the	
  limits	
  of	
  your	
  current	
  design
You’ve	
  run	
  experiments	
  inspired	
  by	
  blog	
  posts
But…
Only	
  1	
  out	
  of	
  5	
  tests	
  win.
4	
  out	
  of	
  5	
  tests	
  
cost	
  you	
  money.
$
There	
  are	
  no	
  magic	
  bullets	
  or	
  pixie	
  dust
You	
  need	
  to	
  double	
  your	
  win	
  rate.
Tools
Usability	
  Tes5ng
hEp://usertes0ng.com
Mouse	
  Tracking
hEp://crazyegg.com
On-­‐Page	
  Surveying
hEp://qualaroo.com
Live	
  Chat
hEp://olark.com
Long	
  Form	
  Surveying
hEp://surveymonkey.com
5	
  tac5cal	
  5ps	
  to	
  double	
  your	
  win	
  rate
1.	
  Find	
  out	
  why	
  people	
  don’t	
  convert	
  
2.	
  Use	
  the	
  exact	
  words	
  customers	
  do	
  
3.	
 ...
Discover	
  what	
  informa5on	
  is	
  missing
Ask	
  people	
  when	
  they	
  try	
  to	
  leave
Email	
  a	
  survey
Ask	
  people	
  why	
  they	
  cancelled
Get	
  feedback	
  right	
  when	
  people	
  cancel
1.	
  Find	
  out	
  why	
  people	
  don’t	
  convert	
  
2.	
  Use	
  the	
  exact	
  words	
  customers	
  do	
  
3.	
 ...
Learn	
  what	
  frustrates	
  them
Find	
  out	
  how	
  they	
  describe	
  your	
  product
Test	
  very	
  specific	
  call	
  to	
  ac5ons	
  (CTAs)
1.	
  Find	
  out	
  why	
  people	
  don’t	
  convert	
  
2.	
  Use	
  the	
  exact	
  words	
  customers	
  do	
  
3.	
 ...
“A solid test hypothesis is an informed solution to
a real problem – not an arbitrary guess. The more
research and data yo...
Answer	
  key	
  ques5ons	
  for	
  each	
  experiment
1.	
  Which	
  top	
  metric	
  are	
  we	
  trying	
  to	
  improv...
Priori5ze	
  your	
  experiments
hEp://kiss.ly/abpie
1.	
  Find	
  out	
  why	
  people	
  don’t	
  convert	
  
2.	
  Use	
  the	
  exact	
  words	
  customers	
  do	
  
3.	
 ...
Waste	
  less	
  code	
  
!
Small	
  experiments	
  help	
  you	
  
focus	
  on	
  learning	
  instead	
  of	
  
wri0ng	
 ...
Know	
  what	
  works	
  
!
Small	
  experiments	
  help	
  you	
  
isolate	
  and	
  understand	
  what	
  
impacts	
  co...
Validate	
  your	
  ideas	
  
!
Small	
  experiments	
  help	
  you	
  
validate	
  assump0ons	
  about	
  
your	
  produc...
1.	
  Find	
  out	
  why	
  people	
  don’t	
  convert	
  
2.	
  Use	
  the	
  exact	
  words	
  customers	
  do	
  
3.	
 ...
Con5nuous	
  Experimenta5on	
  
Small,	
  measurable	
  changes.	
  
Keeps	
  us	
  honest.	
  
Prevents	
  us	
  from	
  ...
One	
  more	
  thing...
Learn	
  more…	
  hEp://grow.kissmetrics.com/op0con
NEW!	
  KISSmetrics	
  Customizable	
  A/B	
  ...
THANK	
  YOU
Hiten	
  Shah	
  
@hnshah
Hiten	
  Shah	
  
Co-­‐Founder	
  
KISSmetrics	
  
@hnhshah	
  
Upcoming SlideShare
Loading in …5
×

How to Increase Your Testing Success by Combining Qualitative and Quantitative Data

985 views

Published on

Hiten Shah, President and Co-Founder, KISSmetrics and Crazy Egg

The majority of A/B tests that you run end up failing. Wouldn't it be great if you could increase your chance of success?

In this session Hiten Shah, President and Co-Founder of KISSmetrics and Crazy Egg provides a framework and examples of how to increase your success rate by using both qualitative and quantitative tactics. Learn how to design great experiments by understanding more about your visitors, users and customers.

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
985
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
15
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

How to Increase Your Testing Success by Combining Qualitative and Quantitative Data

  1. 1. How  to    Increase  Your  Tes0ng  Success  by   Combining  Qualita0ve  and  Qualita0ve  Data Hiten  Shah,  KISSmetrics
  2. 2. Hiten  Shah   Co-­‐Founder   KISSmetrics   @hnhshah  
  3. 3. Hiten  Shah   Co-­‐Founder   KISSmetrics   @hnhshah  
  4. 4. There’s  a  big  problem  with  A/B  tes5ng!
  5. 5. You’ve  tested  the  color  of  your  bu?ons
  6. 6. You’ve  watched  your  compe5tors
  7. 7. You’ve  tried  tes5ng  everyone’s  opinion
  8. 8. You’ve  hit  the  limits  of  your  current  design
  9. 9. You’ve  run  experiments  inspired  by  blog  posts
  10. 10. But…
  11. 11. Only  1  out  of  5  tests  win. 4  out  of  5  tests   cost  you  money. $
  12. 12. There  are  no  magic  bullets  or  pixie  dust
  13. 13. You  need  to  double  your  win  rate.
  14. 14. Tools
  15. 15. Usability  Tes5ng hEp://usertes0ng.com
  16. 16. Mouse  Tracking hEp://crazyegg.com
  17. 17. On-­‐Page  Surveying hEp://qualaroo.com
  18. 18. Live  Chat hEp://olark.com
  19. 19. Long  Form  Surveying hEp://surveymonkey.com
  20. 20. 5  tac5cal  5ps  to  double  your  win  rate
  21. 21. 1.  Find  out  why  people  don’t  convert   2.  Use  the  exact  words  customers  do   3.  Start  with  a  hypothesis  then  priori5ze   4.  Create  experiments  that  are  small   5.  Con5nuously  experiment
  22. 22. Discover  what  informa5on  is  missing
  23. 23. Ask  people  when  they  try  to  leave
  24. 24. Email  a  survey
  25. 25. Ask  people  why  they  cancelled
  26. 26. Get  feedback  right  when  people  cancel
  27. 27. 1.  Find  out  why  people  don’t  convert   2.  Use  the  exact  words  customers  do   3.  Start  with  a  hypothesis  then  priori5ze   4.  Create  experiments  that  are  small   5.  Con5nuously  experiment
  28. 28. Learn  what  frustrates  them
  29. 29. Find  out  how  they  describe  your  product
  30. 30. Test  very  specific  call  to  ac5ons  (CTAs)
  31. 31. 1.  Find  out  why  people  don’t  convert   2.  Use  the  exact  words  customers  do   3.  Start  with  a  hypothesis  then  priori5ze   4.  Create  experiments  that  are  small   5.  Con5nuously  experiment
  32. 32. “A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the be er it will be.”
 ! — Michael Aagaard, h p://kiss.ly/abthypo
  33. 33. Answer  key  ques5ons  for  each  experiment 1.  Which  top  metric  are  we  trying  to  improve?   2.  What  can  you  anchor  your  metric  against?   3.  What  is  your  hypothesis  about  why  this  metric  is  at  x%?   4.  What  do  we  expect  from  this  experiment?   5.  What  was  the  result  and  what  did  we  learn  from  it? hEp://kiss.ly/6qsab
  34. 34. Priori5ze  your  experiments hEp://kiss.ly/abpie
  35. 35. 1.  Find  out  why  people  don’t  convert   2.  Use  the  exact  words  customers  do   3.  Start  with  a  hypothesis  then  priori5ze   4.  Create  experiments  that  are  small   5.  Con5nuously  experiment
  36. 36. Waste  less  code   ! Small  experiments  help  you   focus  on  learning  instead  of   wri0ng  code.
  37. 37. Know  what  works   ! Small  experiments  help  you   isolate  and  understand  what   impacts  conversions.
  38. 38. Validate  your  ideas   ! Small  experiments  help  you   validate  assump0ons  about   your  product.
  39. 39. 1.  Find  out  why  people  don’t  convert   2.  Use  the  exact  words  customers  do   3.  Start  with  a  hypothesis  then  priori5ze   4.  Create  experiments  that  are  small   5.  Con5nuously  experiment
  40. 40. Con5nuous  Experimenta5on   Small,  measurable  changes.   Keeps  us  honest.   Prevents  us  from  breaking  things. hEp://kiss.ly/etsycont Dan  McKinley   @mcfunley   @etsy
  41. 41. One  more  thing... Learn  more…  hEp://grow.kissmetrics.com/op0con NEW!  KISSmetrics  Customizable  A/B  Tes5ng  Report
  42. 42. THANK  YOU Hiten  Shah   @hnshah
  43. 43. Hiten  Shah   Co-­‐Founder   KISSmetrics   @hnhshah  

×