Data + Product: Cornell Tech Lecture, Feb 11, 2016
โขDownload as PPTX, PDFโข
0 likesโข389 views
These talk provides a top down overview of how to incorporate data driven decision making and optimization into your product development process. The talk was delivered as part of Cornell Tech's Product Management course.
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Data + Product: Cornell Tech Lecture, Feb 11, 2016
- 1. Product + Data Cornell Tech - February 11, 2016
- 2. Who, Where, and What? Why? How? (The 5 Wโs of Product Data)
- 4. Behavioral Data What did they do? Raw Data: Events Aggregates: Sessions / Visits Metrics: Time in App, Visits with Search, Bounces, Conversions
- 5. (2/9 3:10:15, โHomeโ) (2/9 3:10:18, โSearch Bar Hoverโ) (2/9 3:10:25, โSearchโ {query: โbananasโ, page: 1}) (2/9 3:10:38, โSearchโ {query: โbananasโ, page: 2})
- 6. Audience Data Who are they? Where are they from? Raw Data: Referrers / Demographics Aggregates: Campaigns / Geo regions Metrics: Adwords ROI, Gender dist.
- 7. Core Business Data What did they buy? Raw Data: Transactions, Videos Posted Aggregates: Total Revenue Metrics: Revenue / User, Hours of Video Content
- 8. Everything Else Operational Data Survey Data Support Data Third Party Data Net Promoter Scores User Testing
- 9. Reflect: What types of data are necessary to describe your product?
- 10. Why?
- 11. Example KPIs Lots of Users Lots of Engagement Lots of Money
- 12. Ideal: User Test Everyone
- 13. Funnel Conversions (never 100%) View Item Add to Cart (15%) Purchase (4%)
- 14. Product Loops
- 15. Referrer Conversion Rate google.com facebook.com techcrunch.com Drill Downs 8% 2% 0.01%
- 17. Cohorts
- 18. Reflect: Pick 3 KPIs for your product, a core funnel, and an early adopter customer segment.
- 19. How?
- 21. Dashboards
- 22. Metric Man!
- 23. Reflect: Design a dashboard for your product.
- 24. Studio
- 25. Create your own dashboard (30 minutes)
- 26. Dashboard review What story are you telling? Rank order your KPIs. Discuss redundancy - can anything be removed? Should anything be segmented?
Editor's Notes
- The ultimate goal of analyzing your product through the lens of data is for improvement and optimization. This begins with data that describes your product directly and with the data that surrounds it: whoโs using it, where did they come from, and what did they do? Once these fundamental measurements are in place, and once youโve shipped your MVP, youโll start to measure some basic aggregates and ask the question: Why arenโt my numbers higher? Finally, this insight should lead to action in the form of how to make improvements to your product, and how to best measure these improvements. Weโll also talk about effective dashboarding and other ways to socialize data within your startup, product group, or multinational enterprise.
- Letโs start by digging into the nuts & bolts of product data. There are many types of product data, and weโll start here by covering three of the fundamentals: behavioral data (how people interact with your product), audience data (where they come from and who your users are), and core customer data (what they bought, what they published)
- The raw construct of behavioral data is the event. Events include a timestamp, a user identifier, event name, and a set of key / value properties. On the web, events are either tied to page views or are otherwise triggered via JavaScript - generally in the form of a button click, form submission, or other user-initiated action. For mobile apps or connected devices, events are less structured and are defined on a problem-specific basis. Events are generally aggregated together to form a contiguous series called a session or visit. In traditional web contexts, this is defined as a sequence with no two events more than 30 minutes apart. Sessions (or visits) can take on different definitions in other contexts.
- Hereโs an example of a series of four contiguous events of a userโs search stream. The visit starts with a visit to the homepage, records when the user enters their cursor in the search bar, and then logs resulting search views. Once youโve formed these sorts of aggregates, you can start to compute statistics from them. At Etsy, one of the first things I worked on was improving search, and I started by computing some basic search visit stats. On average, users searched through four pages of results after submitting their query. This was much different than stats I had seen at Google, where 90% of all queries click on the first result on the first page, and fewer than 2% of queries ever make it to the second page. This ended up changing how we thought about search on Etsy: we focused more on discovery and less on optimizations to find โthe single best resultโ. Events are limited in several ways. They canโt capture everything - the search example above doesnโt record which button they clicked on to get to page 2, for example. Event models can get very complex, so donโt attempt to measure everything. Events naturally describe user initiated actions, but are hard to user for โbackgroundโ events such as a declined credit card. Aggregates are hard to capture with events. Consider for example YouTube and a metric like โtotal videos on platformโ. From an event perspective, videos can be either added or removed - tallying this though in an event-based context can be quite tricky. Events are write-once logs and are not authoritative; itโs quite common to have bugs in your deployment, and itโs generally not feasible to go back and fix the data.
- Audience data catalogs who used your product and where they came from. In traditional web contexts, web (or http) requests have a referrer built into the protocol, making the where more easily trackable (although harder with secure http). Additionally, standards have also arisen around campaign tracking that plug into standard tools such as Google Analytics (e.g. params such as &utm_campaign=foo_bar). For mobile contexts, source tracking is harder and standards donโt really exist - in fact, app marketplaces such as the App Store can make source tracking very hard. Demographic targeting today is fairly standardized and IP-based measures offer decent resolution as a starting point to understand geo, income levels, or other basics. Genders or other domain-specific demographics can sometimes be best understood via surveys or questions that are baked into registration flows. Understanding your audience and underlying constituents is critical to understand your productโs performance. At Etsy, a large fraction of traffic came from Google search, and this traffic furthermore converted quite highly. Yet this traffic was highly variable, and every year when Google would tweak their search rankings, Etsyโs Google traffic would change significantly.
- Core business data is the most varied and contains your bottom line mission critical metrics: transactions, content created, items listed, etc. Core business data is authoritative by definition, and reflects the reality of your product and business.
- The world of product-related data is vast, and there are no rules. Pretty much anything can impact how users interact with your product: slow search response times can impact conversion rates, customer support data can indicate buggy or confusing aspects of your product experience.
- Once your product is properly measured and data is collected, you should be able to start measuring performance. This process should begin with defining your key performance indicators, and then diving into the data to better understand why your numbers are where they are.
- Three example KPIs. In an ideal context, youโll establish your KPIs, roll out your product, and your KPIs will all look great. In practice, your KPIs may not all look that great, and this is where youโll want to start to think about Why.
- As a thought experiment, an ideal solution to understanding why people behave the way the do would be to (1) spy on all your users when theyโre interacting with your product in the comfort of their own home, and then (2) collect a complete set of results that describe everything that they did. In reality, product data is lossy: it doesnโt reconstruct the full picture of oneโs full set of interactions, so we need to resort to other means for inferring Why.
- If you have an e-commerce shop, your goal is to get people to buy stuff. People will visit your website, people will add items to their cart, and some people will buy. The number of people who consider buying will also be larger than the actual number who complete this funnel. So in order to understand why your conversion rate is, say, 4%, itโs necessary to break things down in terms of the steps required to arrive there. Standard funnel analysis does this by analyzing primarily behavioral event data, stitching across sessions, and computing drop offs at logical points.
- Once youโve optimized your funnel, you want to step back and look holistically around repeat purchase, continued engagement, and optimize for โproduct loopsโ.
- When diving into your KPIs, youโll want to explore various dimensions that will correlate with performance. If a customer comes to Google and types โAmerican flag shortsโ, they generally have a pretty good idea of what theyโre looking for, and upon entering your site, theyโre well on their way to completing a purchase. In contrast, if a customer reads about Etsy for the first time in the New York Times, theyโll arrive at the homepage with little to no purchase intent.
- Another effective way of understanding whatโs working what isnโt, is to ask the question: whoโs working and whoโs not. Segmentation is best leveraged when youโre able to apply your intuition about the product and business, and overlay on top of your KPIs or other usage metrics. Identifying early adopters is an exercise in segmentation. You should have a strong hypothesis about your prototypical early adopter; your early MVPs should be optimized around this, and you should measure directly. As an example, Wazeโs early adopters who folks who wanted to avoid speed traps, and this segment consisted of users who used cop-locating functionality. Understanding engagement of this segment is critical in understanding MVP success. As your product grows and evolves, youโll start to explore other segments of your user base and hone in on these to identify road bumps or other causes of customer dissatisfaction.
- Cohorts are a special type of segment that are parameterized by time, and the cohort analysis chart above plots retention over the life of your product. The chart above represents a standard cohort representation. The y-axis shows time intervals wherein people completed an action (first visited, registered, purchased) and the x-axis shows downstream retention over this time period. If your product is improving over time, youโll see stronger gradation patterns across the bottom section of the chart, indicating increased retention across more recent cohorts.
- Defining your KPIs are only as good as your ability to socialize them within your team and company. If youโre able to align your team around a common set of goals, then discussions, arguments, and disagreements around requisite product improvements become much more straightforward.
- Given two population of individuals that differ only in a single variable, an A/B test can directly measures changes in response and therefore make causal inferences over the controlled variable. Itโs an invaluable tool for both understanding why your product works the way it does, as well as for improving and optimizing how it works. A/B testing is expensive as it can require a large population of users (which you may not have) and running multiple tests at the same time can sometimes introduce complications as well. Itโs important to check valid statistics before running a controlled experiment; if your user base is small or if youโre optimizing very small conversion rates, you may not have the luxury of being able to A/B test your improvements.
- Dashboards are an accessible format for visualizing and socializing your KPIs. They provide a point of reference for your team to focus on, and should be organized in a hierarchical format that starts with high level KPIs and then drills down into specific dimensions and customer segments.
- Every startup needs a Metric Man: a voice of data that reminds everyone of the cold hard facts.