Lean Analytics Cycle

Lean Analytics Lean Product & Lean UX Silicon Valley - July 2014 Hiten Shah hnshah@gmail.com Download the slides: h p://kiss.ly/leanprod2014

Continuous Learning & Improvement

Don’t know where to start?

Wondering if you’re working on the right things?

Or, you just want to hear me babble!

HTTP://KISS.LY/LEANLYTICSBOOK

Lean Analytics helps you Optimize for Learning

Lean Analytics Cycle AVINASH KAUSHIK HTTP://KISS.LY/LEANAC

Metrics → Hypothesis → Experiment → Act

HTTP://HELLOBAR.COM

What is Hello Bar?

Metrics Figure out what to improve Hypothesis Make an educated guess Experiment Test your guess Act Decide what’s next

Our funnel showed a low installation rate Hello Bar Funnel Metrics Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 15068 1868 716 688 398 % prev step 12.40% 38.33% 96.09% 57.85% % drop off -87.60% -61.67% -3.91% -42.15% % of visitors 100% 12.40% 4.75% 4.57% 2.64%

We achieved a 40% increase for our installa2on rate with just this one ﬁrst experiment. The Lean Analytics Cycle At Work

Original Hello Bar installation screen

Find out why people don’t install Hello Bar

Didn’t Meet My Expectations

I Had Trouble Installing It

I Don’t Have A Website

Not Ready To Install

Other

Hello Bar Experiment Hypothesis If more installa2on op2ons are given, then more people will add Hello Bar to their website due to the need for a WordPress plugin and ability to email the code to someone else.

Created a new ﬂow to test

I can install code myself.

41.67% responded with “WordPress Plugin”

25% responded with “email my developer / web designer”

I need help.

Full funnel metrics for the experiment Variation Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7456 906 342 329 276 % prev step 12.15% 37.75% 96.20% 83.89% % drop off -87.85% -62.25% -3.80% -16.11% % of visitors 100% 12.15% 4.59% 4.41% 3.70% Control Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7469 925 353 341 197 % prev step 12.38% 38.16% 96.60% 57.77% % drop off -87.62% -61.84% -3.40% -42.23% % of visitors 100% 12.38% 4.73% 4.57% 2.64%

Resulted in a 30% installation rate increase HTTP://ISVALID.ORG Experiment resulted in a 40% improvement

ACer a dozen more experiments, our installa2on rate is now hovering between 5% and 7% That’s an addi;onal 89% increase!!! The Lean Analytics Cycle At Work

Metrics → Hypothesis → Experiment → Act How You Can Put It Into Practice

Metrics Figure out what to improve

“What gets measured, gets managed.” Peter Drucker

Baselines DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA

Funnels KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

SaaS Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

eCommerce Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

Revenue BUFFER BAREMETRICS HTTP://KISS.LY/BUFFERBARE

Spreadsheets and Calculations HELLOBAR HTTP://HELLOBAR.COM Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Created Second Bar Totals 15068 1868 716 688 398 193 % prev step 12.40% 38.33% 96.09% 57.85% 48.49% % drop off -87.60% -61.67% -3.91% -42.15% -51.51% % of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%

Hypothesis Make an educated guess

“A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the be er it will be.”  — Michael Aagaard, h p://kiss.ly/abthypo

Research Tools & Tac;cs

Customer Development HTTP://WWW.CUSTDEV.COM

Jobs To Be Done #JTBD HTTP://JOBSTOBEDONE.ORG

Usability Testing HTTP://USERTESTING.COM

Live Chat HTTP://OLARK.COM

Mouse Tracking HTTP://CRAZYEGG.COM

Long Form Surveying HTTP://SURVEYMONKEY.COM

On-Page Surveying HTTP://QUALAROO.COM

Personalized Messaging HTTP://INTERCOM.IO

Email a survey to people HTTP://SURVEYMONKEY.COM

Ask people why they canceled HTTP://SURVEYMONKEY.COM

Get feedback right when they cancel

Exit Surveys HTTP://QUALAROO.COM

Discover what’s missing HTTP://QUALAROO.COM

Find out what words people use to describe your product HTTP://QUALAROO.COM

Learn about people’s expectations HTTP://INTERCOM.IO

Learn what frustrates people HTTP://INTERCOM.IO

More actionable reading… QUALITATIVE DATA HTTP://KISS.LY/QUALABT

Experiment Hypothesis Formula DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA

Experiment Test your guess

Only 1 out of 5 tests win. 4 out of 5 tests cost you money. $

1) Which top metric are we trying to improve? 2) What can you anchor your metric against? 3) What is your hypothesis about why this metric is at x%? 4) What do we expect from this experiment? 5) What was the result and what did we learn from it? HTTP://KISS.LY/6QSAB Answer key questions for each experiment

Prioritize your experiments HTTP://KISS.LY/ABPIE

Don’t copy “best practices” blindly

Learn what works best for you

Act Decide what’s next

Explore the results and make data informed decisions. THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM

KISSmetrics A/B Testing Report KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

We know that people love Google Analytics

KISSmetrics Experiment Hypothesis If people authen2cate with Google, then more people will sign up due to their exis2ng familiarity with Google Analy2cs and alignment with our winning headline.

KISSmetrics Homepage Test Control

KISSmetrics Homepage Test Variation

KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

KISSmetrics Homepage New Control

Inspiration

KISSmetrics Homepage New Test Variation

KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

Growth! HTTP://KISSMETRICS.COM

Hiten Shah hnshah@gmail.com @hnshah You will get all you want in life if you help enough other people get what they want. Zig Ziglar Download the slides: h p://kiss.ly/leanprod2014