Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Milion Dollar Impact Through Metrics, Analytics & A/B Testing

This is an updated version of the talk I gave at Product Camp ATL 2017.

Atlanta doesn't talk enough about data analytics and A/B testing. I was able to follow my product's performance on a product scorecard built on (free) analytics tools to identify key areas of improvement. Once identified, I conducted A/B testing on potential fixes to find the best performing change that resulted in real revenue impact. This session is a case study on what I did and how you can do the same for your product.

Note: I presented this session last year and it helped someone get a promotion at work! So, I'll be presenting an updated version of it in 2017.

  • Be the first to comment

Milion Dollar Impact Through Metrics, Analytics & A/B Testing

  1. 1. Million Dollar Impact Through Metrics, Analytics, & A/B Testing
  2. 2. Hello! I am Azhar Bande-Ali Software Engineer turned PM with a passion for people and data You can find me at: @AzharBA
  3. 3. In this talk ◎ Discuss web products at scale ◎ Narrow approach due to time constraints ◎ Principles can be adapted to your world Feel free to ask questions
  4. 4. Metrics Define your measurement criteria
  5. 5. Top Down Metrics Measure what your business cares about. Everything else is vanity.
  6. 6. Place your screenshot here E-commerce Website Improve yearly revenue from online sales
  7. 7. Metrics for growing revenue
  8. 8. Conversion Rate Metrics for growing revenue Drop-off Rate Return Rate
  9. 9. Conversion Rate % of unique sessions with a submitted order Metrics for growing revenue Drop-off Rate For every page in the ordering flow: % of sessions that don’t graduate to next step Return Rate % of new users who repeat order Compare each metric to your target/goal and industry average!
  10. 10. Analytics Measure your metrics
  11. 11. OR
  12. 12. Getting started with Analytics ◎ Needs code change - development cost ◎ Measure: ○ Conversion rate of funnels and ○ Drop off rates on pages ◎ Event tracking for more intricate analysis
  13. 13. Marry analytics to business goals Conversion Rate Drop off Rate Return Rate
  14. 14. Measure Frameworks ◎ Funnels ◎ AIDA ○ Awareness ○ Interest ○ Desire ○ Action
  15. 15. Measure Frameworks ◎ Funnels ◎ AIDA ○ Attention ○ Interest ○ Desire ○ Action Events
  16. 16. Drop-offs
  17. 17. ROI
  18. 18. How to get buy in from your boss and partners
  19. 19. $1,085,136 How to get buy in from your boss and partners
  20. 20. A/B Test Validate your hypothesis
  21. 21. Hypothesis Making “Continue” button more prominent will improve the checkout conversion rate.
  22. 22. Variant 1 Remove “Cancel Order” Button
  23. 23. Variant 2 Remove “Cancel Order” button and rename “Continue” to “Checkout”
  24. 24. And the winner is..
  25. 25. And the winner is.. ..neither!
  26. 26. Truth about A/B testing Diminishing Return Even a successful A/B test variant will eventually start failing for a variety of reasons. No Winners It is possible that none of your variants cause a big enough change to be declared winner Success Rate According to KissMetrics, only 1 out of 8 A/B tests have a valid winner Significance Sample being tested on should be big enough to represent population Control Group Always show a ‘no change’ state to a control group to compare understand what would happen if you did nothing Representation The test should be conducted on the same group of users that are a sample of the actual user set
  27. 27. Next Steps Org Support Ensure that your organization recognizes the value of the opportunity enough to prioritize it Iterate Be ready to come up with several hypothesis about which features create value for users Failure Even the best teams have a 12.5% chance of success. Know that it isn’t easy and you’ll fail a lot
  28. 28. Thanks! Any questions? You can find me at: @AzharBA pcampatl@azharb.com

    Be the first to comment

    Login to see the comments

  • safalkapoor

    Mar. 29, 2019

This is an updated version of the talk I gave at Product Camp ATL 2017. Atlanta doesn't talk enough about data analytics and A/B testing. I was able to follow my product's performance on a product scorecard built on (free) analytics tools to identify key areas of improvement. Once identified, I conducted A/B testing on potential fixes to find the best performing change that resulted in real revenue impact. This session is a case study on what I did and how you can do the same for your product. Note: I presented this session last year and it helped someone get a promotion at work! So, I'll be presenting an updated version of it in 2017.

Views

Total views

281

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

11

Shares

0

Comments

0

Likes

1

×