0
Lean Analytics
Lean Product & Lean UX Silicon Valley - July 2014
Hiten Shah
hnshah@gmail.com
Download the slides: h p://ki...
Continuous Learning & Improvement
Don’t know where to start?
Wondering if you’re working on the right things?
Or, you just want to hear me babble!
HTTP://KISS.LY/LEANLYTICSBOOK
Lean Analytics helps you Optimize for Learning
Lean Analytics Cycle
AVINASH KAUSHIK HTTP://KISS.LY/LEANAC
Metrics	
  →	
  Hypothesis	
  →	
  Experiment	
  →	
  Act
HTTP://HELLOBAR.COM
What is Hello Bar?
 Metrics	
  
	
  Figure	
  out	
  what	
  to	
  improve	
  
	
  Hypothesis	
  
	
  Make	
  an	
  educated	
  guess	
  
	
 ...
Our funnel showed a low installation rate
Hello Bar Funnel Metrics
Steps Visitors Homepage
URL Entries
Created
First Bar
C...
We	
  achieved	
  a	
  40%	
  increase	
  for	
  our	
  
installa2on	
  rate	
  with	
  just	
  this	
  one	
  
first	
  ex...
Original Hello Bar installation screen
 Metrics	
  
	
  Figure	
  out	
  what	
  to	
  improve	
  
	
  Hypothesis	
  
	
  Make	
  an	
  educated	
  guess	
  
	
 ...
Find out why people don’t install Hello Bar
Didn’t Meet My Expectations
I Had Trouble Installing It
I Don’t Have A Website
Not Ready To Install
Other
Hello Bar Experiment Hypothesis
If	
  more	
  installa2on	
  op2ons	
  are	
  given,	
  
then	
  more	
  people	
  will	
 ...
 Metrics	
  
	
  Figure	
  out	
  what	
  to	
  improve	
  
	
  Hypothesis	
  
	
  Make	
  an	
  educated	
  guess	
  
	
 ...
Created a new flow to test
I can install code myself.
41.67% responded with “WordPress Plugin”
25% responded with “email my developer / web designer”
I need help.
 Metrics	
  
	
  Figure	
  out	
  what	
  to	
  improve	
  
	
  Hypothesis	
  
	
  Make	
  an	
  educated	
  guess	
  
	
 ...
Full funnel metrics for the experiment
Variation
Steps Visitors Homepage
URL Entries
Created
First Bar
Completed
Registrat...
Resulted in a 30% installation rate increase
HTTP://ISVALID.ORG
Experiment resulted in a 40% improvement
ACer	
  a	
  dozen	
  more	
  experiments,	
  our	
  
installa2on	
  rate	
  is	
  now	
  hovering	
  
between	
  5%	
  an...
Metrics	
  →	
  Hypothesis	
  →	
  Experiment	
  →	
  Act
How You Can Put It Into Practice
Metrics	
  
Figure	
  out	
  what	
  to	
  improve
“What gets measured, gets managed.” Peter Drucker
Baselines
DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
Funnels
KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
SaaS Funnel
KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
eCommerce Funnel
KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
Revenue
BUFFER BAREMETRICS HTTP://KISS.LY/BUFFERBARE
Spreadsheets and Calculations
HELLOBAR HTTP://HELLOBAR.COM
Steps Visitors Homepage
URL Entries
Created
First Bar
Completed...
Hypothesis	
  
Make	
  an	
  educated	
  guess
“A solid test hypothesis is an informed solution to
a real problem – not an arbitrary guess. The more
research and data yo...
Research	
  
Tools	
  &	
  Tac;cs
Customer Development
HTTP://WWW.CUSTDEV.COM
Jobs To Be Done #JTBD
HTTP://JOBSTOBEDONE.ORG
Usability Testing
HTTP://USERTESTING.COM
Live Chat
HTTP://OLARK.COM
Mouse Tracking
HTTP://CRAZYEGG.COM
Long Form Surveying
HTTP://SURVEYMONKEY.COM
On-Page Surveying
HTTP://QUALAROO.COM
Personalized Messaging
HTTP://INTERCOM.IO
Email a survey to people
HTTP://SURVEYMONKEY.COM
Ask people why they canceled
HTTP://SURVEYMONKEY.COM
Get feedback right when they cancel
Exit Surveys
HTTP://QUALAROO.COM
Discover what’s missing
HTTP://QUALAROO.COM
Find out what words people use to describe your product
HTTP://QUALAROO.COM
Learn about people’s expectations
HTTP://INTERCOM.IO
Learn what frustrates people
HTTP://INTERCOM.IO
More actionable reading…
QUALITATIVE DATA HTTP://KISS.LY/QUALABT
Experiment Hypothesis Formula
DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
Experiment	
  
Test	
  your	
  guess
Only	
  1	
  out	
  of	
  5	
  tests	
  win.
4	
  out	
  of	
  5	
  tests	
  
cost	
  you	
  money.
$
1)	
  Which	
  top	
  metric	
  are	
  we	
  trying	
  to	
  improve?	
  
2)	
  What	
  can	
  you	
  anchor	
  your	
  me...
Prioritize your experiments
HTTP://KISS.LY/ABPIE
Don’t copy “best practices” blindly
Learn what works best for you
Act	
  
Decide	
  what’s	
  next
Explore	
  the	
  results	
  and	
  make	
  
data	
  informed	
  decisions.
THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://K...
KISSmetrics A/B Testing Report
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
We know that people love Google Analytics
KISSmetrics Experiment Hypothesis
If	
  people	
  authen2cate	
  with	
  Google,	
  
then	
  more	
  people	
  will	
  sig...
KISSmetrics Homepage Test Control
KISSmetrics Homepage Test Variation
KISSmetrics Homepage Test Signup Conversion Rate
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
KISSmetrics Homepage Test Received Data
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
KISSmetrics Homepage New Control
Inspiration
KISSmetrics Homepage New Test Variation
KISSmetrics Homepage Test Signup Conversion Rate
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
KISSmetrics Homepage Test Received Data
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
Growth!
HTTP://KISSMETRICS.COM
 Metrics	
  
	
  Figure	
  out	
  what	
  to	
  improve	
  
	
  Hypothesis	
  
	
  Make	
  an	
  educated	
  guess	
  
	
 ...
Hiten Shah
hnshah@gmail.com
@hnshah
You will get all you want in life if you help enough
other people get what they want.
...
Lean Analytics Cycle
Lean Analytics Cycle
Lean Analytics Cycle
Lean Analytics Cycle
Lean Analytics Cycle
Lean Analytics Cycle
Lean Analytics Cycle
Lean Analytics Cycle
Lean Analytics Cycle
Lean Analytics Cycle
Upcoming SlideShare
Loading in...5
×

Lean Analytics Cycle

115,713

Published on

Published in: Data & Analytics, Technology
1 Comment
281 Likes
Statistics
Notes
  • hi hiten shah ur presentation is really v good
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
115,713
On Slideshare
0
From Embeds
0
Number of Embeds
55
Actions
Shares
0
Downloads
1,443
Comments
1
Likes
281
Embeds 0
No embeds

No notes for slide

Transcript of "Lean Analytics Cycle"

  1. 1. Lean Analytics Lean Product & Lean UX Silicon Valley - July 2014 Hiten Shah hnshah@gmail.com Download the slides: h p://kiss.ly/leanprod2014
  2. 2. Continuous Learning & Improvement
  3. 3. Don’t know where to start?
  4. 4. Wondering if you’re working on the right things?
  5. 5. Or, you just want to hear me babble!
  6. 6. HTTP://KISS.LY/LEANLYTICSBOOK
  7. 7. Lean Analytics helps you Optimize for Learning
  8. 8. Lean Analytics Cycle AVINASH KAUSHIK HTTP://KISS.LY/LEANAC
  9. 9. Metrics  →  Hypothesis  →  Experiment  →  Act
  10. 10. HTTP://HELLOBAR.COM
  11. 11. What is Hello Bar?
  12. 12.  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  13. 13. Our funnel showed a low installation rate Hello Bar Funnel Metrics Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 15068 1868 716 688 398 % prev step 12.40% 38.33% 96.09% 57.85% % drop off -87.60% -61.67% -3.91% -42.15% % of visitors 100% 12.40% 4.75% 4.57% 2.64%
  14. 14. We  achieved  a  40%  increase  for  our   installa2on  rate  with  just  this  one   first  experiment. The Lean Analytics Cycle At Work
  15. 15. Original Hello Bar installation screen
  16. 16.  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  17. 17. Find out why people don’t install Hello Bar
  18. 18. Didn’t Meet My Expectations
  19. 19. I Had Trouble Installing It
  20. 20. I Don’t Have A Website
  21. 21. Not Ready To Install
  22. 22. Other
  23. 23. Hello Bar Experiment Hypothesis If  more  installa2on  op2ons  are  given,   then  more  people  will  add  Hello  Bar   to  their  website  due  to  the  need  for  a   WordPress  plugin  and  ability  to  email   the  code  to  someone  else.
  24. 24.  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  25. 25. Created a new flow to test
  26. 26. I can install code myself.
  27. 27. 41.67% responded with “WordPress Plugin”
  28. 28. 25% responded with “email my developer / web designer”
  29. 29. I need help.
  30. 30.  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  31. 31. Full funnel metrics for the experiment Variation Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7456 906 342 329 276 % prev step 12.15% 37.75% 96.20% 83.89% % drop off -87.85% -62.25% -3.80% -16.11% % of visitors 100% 12.15% 4.59% 4.41% 3.70% Control Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7469 925 353 341 197 % prev step 12.38% 38.16% 96.60% 57.77% % drop off -87.62% -61.84% -3.40% -42.23% % of visitors 100% 12.38% 4.73% 4.57% 2.64%
  32. 32. Resulted in a 30% installation rate increase HTTP://ISVALID.ORG Experiment resulted in a 40% improvement
  33. 33. ACer  a  dozen  more  experiments,  our   installa2on  rate  is  now  hovering   between  5%  and  7%   That’s  an  addi;onal  89%  increase!!! The Lean Analytics Cycle At Work
  34. 34. Metrics  →  Hypothesis  →  Experiment  →  Act How You Can Put It Into Practice
  35. 35. Metrics   Figure  out  what  to  improve
  36. 36. “What gets measured, gets managed.” Peter Drucker
  37. 37. Baselines DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
  38. 38. Funnels KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  39. 39. SaaS Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  40. 40. eCommerce Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
  41. 41. Revenue BUFFER BAREMETRICS HTTP://KISS.LY/BUFFERBARE
  42. 42. Spreadsheets and Calculations HELLOBAR HTTP://HELLOBAR.COM Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Created Second Bar Totals 15068 1868 716 688 398 193 % prev step 12.40% 38.33% 96.09% 57.85% 48.49% % drop off -87.60% -61.67% -3.91% -42.15% -51.51% % of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%
  43. 43. Hypothesis   Make  an  educated  guess
  44. 44. “A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the be er it will be.”
 — Michael Aagaard, h p://kiss.ly/abthypo
  45. 45. Research   Tools  &  Tac;cs
  46. 46. Customer Development HTTP://WWW.CUSTDEV.COM
  47. 47. Jobs To Be Done #JTBD HTTP://JOBSTOBEDONE.ORG
  48. 48. Usability Testing HTTP://USERTESTING.COM
  49. 49. Live Chat HTTP://OLARK.COM
  50. 50. Mouse Tracking HTTP://CRAZYEGG.COM
  51. 51. Long Form Surveying HTTP://SURVEYMONKEY.COM
  52. 52. On-Page Surveying HTTP://QUALAROO.COM
  53. 53. Personalized Messaging HTTP://INTERCOM.IO
  54. 54. Email a survey to people HTTP://SURVEYMONKEY.COM
  55. 55. Ask people why they canceled HTTP://SURVEYMONKEY.COM
  56. 56. Get feedback right when they cancel
  57. 57. Exit Surveys HTTP://QUALAROO.COM
  58. 58. Discover what’s missing HTTP://QUALAROO.COM
  59. 59. Find out what words people use to describe your product HTTP://QUALAROO.COM
  60. 60. Learn about people’s expectations HTTP://INTERCOM.IO
  61. 61. Learn what frustrates people HTTP://INTERCOM.IO
  62. 62. More actionable reading… QUALITATIVE DATA HTTP://KISS.LY/QUALABT
  63. 63. Experiment Hypothesis Formula DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
  64. 64. Experiment   Test  your  guess
  65. 65. Only  1  out  of  5  tests  win. 4  out  of  5  tests   cost  you  money. $
  66. 66. 1)  Which  top  metric  are  we  trying  to  improve?   2)  What  can  you  anchor  your  metric  against?   3)  What  is  your  hypothesis  about  why  this  metric  is  at  x%?   4)  What  do  we  expect  from  this  experiment?   5)  What  was  the  result  and  what  did  we  learn  from  it? HTTP://KISS.LY/6QSAB Answer key questions for each experiment
  67. 67. Prioritize your experiments HTTP://KISS.LY/ABPIE
  68. 68. Don’t copy “best practices” blindly
  69. 69. Learn what works best for you
  70. 70. Act   Decide  what’s  next
  71. 71. Explore  the  results  and  make   data  informed  decisions. THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM
  72. 72. KISSmetrics A/B Testing Report KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  73. 73. We know that people love Google Analytics
  74. 74. KISSmetrics Experiment Hypothesis If  people  authen2cate  with  Google,   then  more  people  will  sign  up  due  to   their  exis2ng  familiarity  with  Google   Analy2cs  and  alignment  with  our   winning  headline.
  75. 75. KISSmetrics Homepage Test Control
  76. 76. KISSmetrics Homepage Test Variation
  77. 77. KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  78. 78. KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  79. 79. KISSmetrics Homepage New Control
  80. 80. Inspiration
  81. 81. KISSmetrics Homepage New Test Variation
  82. 82. KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  83. 83. KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
  84. 84. Growth! HTTP://KISSMETRICS.COM
  85. 85.  Metrics    Figure  out  what  to  improve    Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess    Act    Decide  what’s  next
  86. 86. Hiten Shah hnshah@gmail.com @hnshah You will get all you want in life if you help enough other people get what they want. Zig Ziglar Download the slides: h p://kiss.ly/leanprod2014
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×