Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Analytics in Action: How to Build Data-Informed Products

206 views

Published on

As a product manager, you probably know specific ways to gather data to inform your product decisions, like the ever-popular A/B test. But as they say, you wouldn't put a round peg in a square hole. What about the times when it doesn't make sense to A/B test, because you have too small a sample size? Do you just do it anyway, because your company's product culture requires that you have those numbers?

Tim Herbig will share his hands-on approach to working with analytics in agile product management. Tim will discuss the line between being data-informed versus data-driven. The audience will leave with an analysis toolkit filled with the right data tools for every scenario. Data validation won't be an issue again.

Published in: Data & Analytics
  • Be the first to comment

Analytics in Action: How to Build Data-Informed Products

  1. 1. How To Build Data-Informed Products Tim Herbig Hannah Flynn With: Moderated by: TO USE YOUR COMPUTER'S AUDIO: When the webinar begins, you will be connected to audio using your computer's microphone and speakers (VoIP). A headset is recommended. Webinar will begin: 11:00 am, PST TO USE YOUR TELEPHONE: If you prefer to use your phone, you must select "Use Telephone" after joining the webinar and call in using the numbers below. United States: +1 (415) 655-0052 Access Code: 985-561-257 Audio PIN: Shown after joining the webinar --OR--
  2. 2. Revulytics gives any software producers deep and actionable insight into who is using their software products and how they are being used, and the out-of-box analytics that enable them to grow incremental revenue, convert and retain customers, and make decisions about licensing and cloud transformation strategies.
  3. 3. Click on the Questions panel to interact with the presenters www.productmanagementtoday.com/webinar-series/analytics-in-action www.projectmanagementupdate.com/webinar-series/analytics-in-action www.businessinnovationbrief.com/webinar-series/analytics-in-action
  4. 4. About Tim Herbig Tim is a product and business leader, as well as a prolific speaker and author of the book ‘Lateral Leadership: A Practical Guide for Agile Product Managers’. Currently, he’s responsible for the overall sales, product, engineering, and marketing efforts behind the leading conversion optimization platform iridion. He previously shaped digital products at XING, Gruner+Jahr, and several smaller startups where he held product leadership roles for more than seven years. Tim also regularly co-organizes the Product Tank Hamburg meetup to promote the local product management community. About Hannah Flynn Hannah went to The University of Chicago, where she majored in Environmental Studies with a concentration in Economics and Policy. She now works with Aggregage on social media strategy and webinar production on sites such as Product Management Today, B2B Marketing Zone, and Customer Experience Update.
  5. 5. @herbigt Insights for Today
  6. 6. Why care about Analytics at all?
  7. 7. To beat the HiPPO
  8. 8. Product Vision & Strategy Product Discovery Product Delivery Analytics Story Mappings User Interviews Webinar: How To Build Data-Informed Products by @herbigt
  9. 9. Product Vision & Strategy Product Discovery Product Delivery Analytics Story Mappings User Interviews Webinar: How To Build Data-Informed Products by @herbigt
  10. 10. Source: uxdesign.cc
  11. 11. 1. Get caught up in micro-conversions. 2. Confuse data with why people are doing things. 3. Leaving interpretation of data to everyone themselves (confirmation bias). 4. Confuse today’s behavior with tomorrow’s potential. Analytics DON’Ts Webinar: How To Build Data-Informed Products by @herbigt
  12. 12. Analytics DOs Webinar: How To Build Data-Informed Products by @herbigt 1. Sanity check your first assumptions about user problems. 2. Use it as a foundation for your hypotheses. 3. Objectify understanding about outcomes. 4. Monitor your product’s ongoing performance.
  13. 13. Analytics Tool Landscape Webinar: How To Build Data-Informed Products by @herbigt Web/App Data A/B Testing Heatmaps DWH/Back- end Tracking Data Viz
  14. 14. Analytics Tool Examples Webinar: How To Build Data-Informed Products by @herbigt Web/App Data ● Used by Product Managers/Analytics Managers ● Mirrors current behavior
  15. 15. Analytics Tool Examples Webinar: How To Build Data-Informed Products by @herbigt A/B Testing ● Used by Product Manager/Conversion Optimization Managers ● Helps to decide which design/version performs better ● Lots of craftsmanship required to nail
  16. 16. Analytics Tool Examples Webinar: How To Build Data-Informed Products by @herbigt Heatmaps ● Used by Product Managers and UX Designers to spot Usability gaps ● Helps to move closer to the why behind analytics data ● Requires few “expertise” to distill information
  17. 17. Analytics Tool Examples Webinar: How To Build Data-Informed Products by @herbigt DWH/Back- end Tracking ● Backbone behind analytics tools ● Used by data analytics managers ● Helpful to double check frontend analytics data ● Allows to unify data from various silos
  18. 18. Analytics Tool Examples Webinar: How To Build Data-Informed Products by @herbigt Data Viz ● Makes analytics data tangible ● Allows to spot similarities between and compare multiple sources ● Unmatched flexibility for building custom dashboards
  19. 19. Great, A/B Test all the things, right?
  20. 20. Why not? Webinar: How To Build Data-Informed Products by @herbigt ● People fancy A/B Testing for the sake of it ● Red flags to watch out for: ○ Lack of hypotheses ○ Used to resolve conflicts → getting consensus and leaving potential on the table for progress ○ Only marginal differences in variations
  21. 21. The biggest A/B Testing Misconceptions + Pitfalls
  22. 22. Sharing Results too early Webinar: How To Build Data-Informed Products by @herbigt ● Tests take time to become valid ● Don’t give in to the nagging of stakeholders to ‘take a look’ ● Instead, make stats and significance calculation visible
  23. 23. Solely relying on calculators Webinar: How To Build Data-Informed Products by @herbigt ● Calculators help you to objectify, but factor in all environmental factors ● E.g. Competing A/B tests, changing discount campaigns changing selection criteria
  24. 24. Bonus: Validity Rule of thumb Webinar: How To Build Data-Informed Products by @herbigt 1. 2 weeks minimum duration to capture ‘seasonal’ fluctuations 2. 500 conversions per variation (ideally 1.000) 3. Significance of >95%
  25. 25. ● Don’t assume you can automatically influence bottom line through micro-conversions at top of the funnel ● We tend to run smaller A/B tests without connecting these KPIs to the overall company goals ● Track and compare cohorts over time to identify impact of test change on e.g. revenue Focussing on wrong KPI Webinar: How To Build Data-Informed Products by @herbigt
  26. 26. Focussing on wrong KPI Webinar: How To Build Data-Informed Products by @herbigt +20% Top of Funnel Doesn’t carry through Increase in Quantity can lead to decrease in Quality = Lower Bottom Line Conversions
  27. 27. Webinar: How To Build Data-Informed Products by @herbigt Travel Group Syndrome
  28. 28. A/B Test Results ≠ User Feedback Webinar: How To Build Data-Informed Products by @herbigt ● ‘Version A won, so our users like this one more!’ ● A/B Tests don’t tell you why people are doing sth. ● It’s the unfortunate discrepancy between what people say they do and what they actually do. ● Check back it’s mid- to long-term impact on your product by conducting some 1:1 user interviews digging into what users actually made them e.g. click more
  29. 29. Don’t A/B Test entire Projects Webinar: How To Build Data-Informed Products by @herbigt ● A/B Tests work for further evolution of existing products ● Don’t help much for new ideas or innovation ● You can’t test business models ● Instead, try quantitative or qualitative experiments to validate individual core hypotheses
  30. 30. Quantitative vs. Qualitative Validation
  31. 31. Webinar: How To Build Data-Informed Products by @herbigt
  32. 32. Validation Example at XING
  33. 33. It’s about indicators not evidence
  34. 34. Low Traffic B2B Environments ● Don’t look for something what’s not there ● Focus on key metrics like weekly active users or NPS ● 10 well executed and recruited user interviews beat obsessing over micro metrics ● Make capturing user feedback a habit and core part of your teams processes
  35. 35. Summary
  36. 36. Thanks! tim@herbigt.com herbigt.com @herbigt herbig.blog/book
  37. 37. Q&A Hannah Flynn With: Moderated by: Director, iridion Linkedin page: linkedin.com/in/herbigt/ Twitter ID: @herbigt Website: herbigt.com Book: herbig.blog/book Tim Herbig Site editor, Product Management Today Linkedin page: linkedin.com/in/hannahmflynn Twitter ID: @prodmgmttoday Email: hannah@aggregage.com Website: productmanagementtoday.com www.productmanagementtoday.com/webinar-series/analytics-in-action www.projectmanagementupdate.com/webinar-series/analytics-in-action www.businessinnovationbrief.com/webinar-series/analytics-in-action

×