SlideShare a Scribd company logo
PM-Summit
The Practice of Data
Driven Products in
Kuaishou
Jay Wang
Former Twitter/Snap Data Scientist
Director of Data Science, Kuaishou
PM-Summit
PM-Summit
A g e n d a
Data Science Overview
Annual plan of a DS team, job responsibility, value
1
Background Knowledge
Some pitfalls, analysis methods
2
DS Applications
Decision support, metrics, AB testing
3
PM-Summit
SELF INTRO
vData analytics & machine learning
vHP Labs: demand forecasting, pricing and
portfolio management
vTwitter: ads prediction and ranking
vStitch fix: e-commerce recommendations
vSnap:video ads and creative tools
PM-Summit
Average DAUs on Kuaishou APP
MAU in overseas market
daily active streamers
Livestreaming user
penetration rate
Total revenue
online marketing services
Total e-commerce GMV
DAU/MAU
pairs of mutual
followers
From 2021 Q2 earning report
PM-Summit
DS TEAM ROADMAP
PM-Summit
IMPACT
Ensure the minimum exposure of small
and medium-sized live streamers,
determine the upper limit of bonus
Improve CTR, or reduce
complaint rate
Scale the analysis by building
dashboards, automating the online
prediction process
Optimize the content review process,
propose UTR metric for measuring
how video consumption leads to
creation
PM-Summit
CUSTOMER GROWTH ANALYTICS
v Increase the supply of customers, make
ads auction more competitive
v Compared to our competitors, we still
have plenty of room to grow daily active
customers
v Increase number of mid-sized customers
to improve the healthiness of ads
marketplace
v Lay the foundation for long-term revenue
growth
Background Analytics Impact
Dashboard: churning
customers
Build a scalable solution
Customer leads-> review ->sales
follow-up
Influence a process
Overview of active customers
Customer life cycle
Customer churn analysis and
churn prediction
Leads generation model
PM-Summit
A g e n d a
Data Science Overview
Annual plan of a DS team, job responsibility, value
1
Background Knowledge
Some pitfalls, analysis methods
2
DS Applications
Decision support, metrics, AB testing
3
PM-Summit
PITFALLS - SELECTION BIAS
v Compare those who saw ads with those who
didn’t see ads on their purchases;
v Response bias when people volunteer to fill out a
questionnaire;
v Only measure survivors in a medical experiment;
selection bias
PM-Summit
PITFALLS - SELECTION BIAS
selection bias
v Add filter reason of“potential_exposure” in
serving pipeline, compare sessions with the filter
reason in base traffic with users who saw the ad;
v Post-stratification
v Introduce mortality rate, test the statistical
significance of mortality rate between treatment
groups
How to correct
v Compare those who saw ads with those who
didn’t see ads on their purchases;
v Response bias when people volunteer to fill out a
questionnaire;
v Only measure survivors in a medical experiment;
PM-Summit
PITFALLS - SIMPSON’S PARADOX
Admission rate by gender
Berkeley admission (1974)
Work example
Admission rate by gender and department
Women tend to apply to departments with intense
competition and low admission rates, while men tend to
apply to departments with less competition and easy to
be admitted (such as engineering school, etc.)
Search penetration rate drops after summer vacation ends.
Might be teenagers are churning from app usage, while
teenagers have high search penetration rate
Mantel–Haenszel correction
Mantel-Haenszel corrected admission rate:
men 32%,women 33%
PM-Summit
PITFALLS - CORRELATION VS CAUSATION
Can we conclude that people are more likely to be tangled in their bedsheets because they eat more
cheese?
Office 365 users that see error messages and experience crashes have lower churn rates
Does that mean that Office 365 should show more error messages or that Microsoft should lower code
quality, causing more crashes to reduce churn rate?
Work example
PM-Summit
Panel data vs cross-sectional data
PM-Summit
WHOLE PICTURE OF SEARCH PRODUCT
PM-Summit
A g e n d a
Data Science Overview
Annual plan of a DS team, job responsibility, value
1
Background Knowledge
Some pitfalls, analysis methods
2
DS Applications
Decision support, metrics, AB testing
3
PM-Summit
DECISION SUPPORT
Who
What
Examples
Executives and business stakeholders
Define, monitor and report the core metrics of a business line
or the company
Interpret changes in core business metrics
Answer high-level questions that impact decision-making
External and internal data, may include competitor and
third-party data sources
What will be next years’ DAU forecast?
Which product lines should the company expand to?
What is the ceiling of video search?
Open-endedness and uncertainty
Trust by top executive leaders
Grasp of internal and external information
Arguments and quality of report
Data
Challenges
PM-Summit
DECISION SUPPORT – METRIC MOVEMENT
PM-Summit
METRIC MOVEMENT – REVENUE
PM-Summit
Peter Drucker (“the founder of modern management”)
If you cannot measure it, you cannot improve it
Goodhart’s Law:
When a measure becomes a target, it ceases to be a good measure
Campbell's Law:
The more any quantitative social indicator is used for social decision-making, the more subject it
will be to corruption pressures and the more apt it will be to distort and corrupt the social
processes it is intended to monitor
QUOTES ON METRICS
Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
PM-Summit
Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
PM-Summit
FROM BUSINESS TO EXPERIMENT METRICS
• Changes in business
• Changes in environment
• Changes in understanding of metrics
Make sure metrics are not easily “gameable”
• e.g. : rats in Seattle
Measurable
Attributable
Sensitive and timely
Modify metrics when
Metrics and incentives
Required by experiment metrics
Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
PM-Summit
PM-Summit
ESTABLISHING CAUSALITY
Eric Colson (Former Netflix VP)
AB testing is the only way to draw causal conclusions.
hierarchy of evidence for establishing causality
Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
PM-Summit
实验平台的成熟度
1. Crawl: Building foundational prerequisites, to compute the summary statistics needed so you can
design, run and analyze a few experiments
2. Walk: Focus on defining standard metrics and getting the organization to run more experiments
(validating instrumentation, running AA test, SRM tests)
3. Run: Running experiments at scale, with comprehensive metrics and decision policy (agreed-on
metrics or OEC metric)
4. Fly: Running experiments as the norm for every change, building institutional memory, feature
teams should be able to analyze experiments on their own (kuaishou and bytedance)
Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
PM-Summit
BE CAUTIONS
User learning effect: search button moved from secondary page to front
page
Independence assumption: sending/receiving messages in social
network, experiment in an isolated world
Combining multiple experiments: 6 experiments with 1% lift during the
quarter, what is the combined effect?
PM-Summit
A g e n d a
Data Science Overview
Annual plan of a DS team, job responsibility, value
1
Background Knowledge
Some pitfalls, analysis methods
2
DS Applications
Decision support, metrics, AB testing
3
PM-Summit
Thank you

More Related Content

What's hot

Predictive Analytics: Business Perspective & Use Cases
Predictive Analytics: Business Perspective & Use CasesPredictive Analytics: Business Perspective & Use Cases
Predictive Analytics: Business Perspective & Use Cases
Cagri Sarigoz
 
Sentiment Analysis: The Marketplace and Providers
Sentiment Analysis: The Marketplace and ProvidersSentiment Analysis: The Marketplace and Providers
Sentiment Analysis: The Marketplace and Providers
Seth Grimes
 
Developing Insights
Developing InsightsDeveloping Insights
Developing Insights
Paul Schumann
 
Is deep learning is a game changer for marketing analytics
Is deep learning is a game changer for marketing analyticsIs deep learning is a game changer for marketing analytics
Is deep learning is a game changer for marketing analytics
BindhuBhargaviTalasi
 
Introduction to MaxDiff Scaling of Importance - Parametric Marketing Slides
Introduction to MaxDiff Scaling of Importance - Parametric Marketing SlidesIntroduction to MaxDiff Scaling of Importance - Parametric Marketing Slides
Introduction to MaxDiff Scaling of Importance - Parametric Marketing Slides
QuestionPro
 
Machine Learning in Retail: Know Your Customers' Customer. See Your Future
Machine Learning in Retail: Know Your Customers' Customer. See Your FutureMachine Learning in Retail: Know Your Customers' Customer. See Your Future
Machine Learning in Retail: Know Your Customers' Customer. See Your Future
BigML, Inc
 
ForresterPredictiveWave
ForresterPredictiveWaveForresterPredictiveWave
ForresterPredictiveWave
Timothy M. Caffrey, MBA
 
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
Jin Young Kim
 
The Future of Applied Marketing Research
The Future of Applied Marketing ResearchThe Future of Applied Marketing Research
The Future of Applied Marketing Research
Kelly Page
 
Predictive analytics in mobility
Predictive analytics in mobilityPredictive analytics in mobility
Predictive analytics in mobility
Ektimo
 
Data Analytics in Azure Cloud
Data Analytics in Azure CloudData Analytics in Azure Cloud
Data Analytics in Azure Cloud
Microsoft Canada
 
predictive analytics
predictive analyticspredictive analytics
predictive analytics
Astha Jagetiya
 
Maxdiff webinar_10_19_10
 Maxdiff webinar_10_19_10 Maxdiff webinar_10_19_10
Maxdiff webinar_10_19_10
QuestionPro
 
Application of predictive analytics
Application of predictive analyticsApplication of predictive analytics
Application of predictive analytics
Prasad Narasimhan
 
The five essential steps to building a data product
The five essential steps to building a data productThe five essential steps to building a data product
The five essential steps to building a data product
Birst
 
Perspectives on Machine Learning
Perspectives on Machine LearningPerspectives on Machine Learning
Perspectives on Machine Learning
Dr. Niren Sirohi
 
Predictive Marketing Analytics
Predictive Marketing AnalyticsPredictive Marketing Analytics
Predictive Marketing Analytics
Lori Fisher
 
Guide: MaxDiff
Guide: MaxDiffGuide: MaxDiff
Guide: MaxDiff
QuestionPro
 
Measuring the Quality of Online Service - Jinyoung kim
Measuring the Quality of Online Service - Jinyoung kimMeasuring the Quality of Online Service - Jinyoung kim
Measuring the Quality of Online Service - Jinyoung kim
Jin Young Kim
 
Business Intelligence & Predictive Analytic by Prof. Lili Saghafi
Business Intelligence & Predictive Analytic by Prof. Lili SaghafiBusiness Intelligence & Predictive Analytic by Prof. Lili Saghafi
Business Intelligence & Predictive Analytic by Prof. Lili Saghafi
Professor Lili Saghafi
 

What's hot (20)

Predictive Analytics: Business Perspective & Use Cases
Predictive Analytics: Business Perspective & Use CasesPredictive Analytics: Business Perspective & Use Cases
Predictive Analytics: Business Perspective & Use Cases
 
Sentiment Analysis: The Marketplace and Providers
Sentiment Analysis: The Marketplace and ProvidersSentiment Analysis: The Marketplace and Providers
Sentiment Analysis: The Marketplace and Providers
 
Developing Insights
Developing InsightsDeveloping Insights
Developing Insights
 
Is deep learning is a game changer for marketing analytics
Is deep learning is a game changer for marketing analyticsIs deep learning is a game changer for marketing analytics
Is deep learning is a game changer for marketing analytics
 
Introduction to MaxDiff Scaling of Importance - Parametric Marketing Slides
Introduction to MaxDiff Scaling of Importance - Parametric Marketing SlidesIntroduction to MaxDiff Scaling of Importance - Parametric Marketing Slides
Introduction to MaxDiff Scaling of Importance - Parametric Marketing Slides
 
Machine Learning in Retail: Know Your Customers' Customer. See Your Future
Machine Learning in Retail: Know Your Customers' Customer. See Your FutureMachine Learning in Retail: Know Your Customers' Customer. See Your Future
Machine Learning in Retail: Know Your Customers' Customer. See Your Future
 
ForresterPredictiveWave
ForresterPredictiveWaveForresterPredictiveWave
ForresterPredictiveWave
 
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
 
The Future of Applied Marketing Research
The Future of Applied Marketing ResearchThe Future of Applied Marketing Research
The Future of Applied Marketing Research
 
Predictive analytics in mobility
Predictive analytics in mobilityPredictive analytics in mobility
Predictive analytics in mobility
 
Data Analytics in Azure Cloud
Data Analytics in Azure CloudData Analytics in Azure Cloud
Data Analytics in Azure Cloud
 
predictive analytics
predictive analyticspredictive analytics
predictive analytics
 
Maxdiff webinar_10_19_10
 Maxdiff webinar_10_19_10 Maxdiff webinar_10_19_10
Maxdiff webinar_10_19_10
 
Application of predictive analytics
Application of predictive analyticsApplication of predictive analytics
Application of predictive analytics
 
The five essential steps to building a data product
The five essential steps to building a data productThe five essential steps to building a data product
The five essential steps to building a data product
 
Perspectives on Machine Learning
Perspectives on Machine LearningPerspectives on Machine Learning
Perspectives on Machine Learning
 
Predictive Marketing Analytics
Predictive Marketing AnalyticsPredictive Marketing Analytics
Predictive Marketing Analytics
 
Guide: MaxDiff
Guide: MaxDiffGuide: MaxDiff
Guide: MaxDiff
 
Measuring the Quality of Online Service - Jinyoung kim
Measuring the Quality of Online Service - Jinyoung kimMeasuring the Quality of Online Service - Jinyoung kim
Measuring the Quality of Online Service - Jinyoung kim
 
Business Intelligence & Predictive Analytic by Prof. Lili Saghafi
Business Intelligence & Predictive Analytic by Prof. Lili SaghafiBusiness Intelligence & Predictive Analytic by Prof. Lili Saghafi
Business Intelligence & Predictive Analytic by Prof. Lili Saghafi
 

Similar to The Practice of Data Driven Products in Kuaishou

Disruptive Innovation
Disruptive InnovationDisruptive Innovation
Disruptive Innovation
Arik Johnson
 
Discover Why Less is More in B2B Research
Discover Why Less is More in B2B ResearchDiscover Why Less is More in B2B Research
Discover Why Less is More in B2B Research
michael115558
 
Intro to Data Analytics with Oscar's Director of Product
 Intro to Data Analytics with Oscar's Director of Product Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product
Product School
 
Voice of the Customer (VOC) Research
Voice of the Customer (VOC) ResearchVoice of the Customer (VOC) Research
Voice of the Customer (VOC) Research
QuestionPro
 
Purchasing & Value Analysis
Purchasing & Value AnalysisPurchasing & Value Analysis
Purchasing & Value Analysis
Thomas Tanel
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Optimizely
 
Operations: Top Reasons for Long Lead Times and What to Do About Them
Operations: Top Reasons for Long Lead Times and What to Do About ThemOperations: Top Reasons for Long Lead Times and What to Do About Them
Operations: Top Reasons for Long Lead Times and What to Do About Them
April Bright
 
How to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your BusinessHow to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your Business
Kissmetrics on SlideShare
 
La Dove Associates -- CRM/Customer Care Consulting Overview
La Dove Associates --  CRM/Customer Care Consulting Overview La Dove Associates --  CRM/Customer Care Consulting Overview
La Dove Associates -- CRM/Customer Care Consulting Overview
LaDove Associates
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'ts
Ramkumar Ravichandran
 
Dat analytics all verticals
Dat analytics all verticalsDat analytics all verticals
Dat analytics all verticals
Digital Analyst Team
 
OKRs KPIs Strategy and Agility.pptx
OKRs KPIs Strategy and Agility.pptxOKRs KPIs Strategy and Agility.pptx
OKRs KPIs Strategy and Agility.pptx
Gtmhub
 
ExpOn Conference Presentation - Sao Paulo, Brazil
ExpOn Conference Presentation - Sao Paulo, BrazilExpOn Conference Presentation - Sao Paulo, Brazil
ExpOn Conference Presentation - Sao Paulo, Brazil
Lima Consulting Group
 
20/10 Vision: Building A 21st Century Market Research Organization
20/10 Vision: Building A 21st Century Market Research Organization20/10 Vision: Building A 21st Century Market Research Organization
20/10 Vision: Building A 21st Century Market Research Organization
Gregory Weiss
 
Competing on analytics
Competing on analyticsCompeting on analytics
Competing on analytics
Greg Seltzer
 
class
classclass
class
abc790027
 
Conversion Whitepaper
Conversion WhitepaperConversion Whitepaper
Conversion Whitepaper
WSI Ensenada
 
The Insider's Guide to Workforce Analytics
The Insider's Guide to Workforce AnalyticsThe Insider's Guide to Workforce Analytics
The Insider's Guide to Workforce Analytics
Visier
 
7 Factors to Consider When Building Your Mobile Strategy
7 Factors to Consider When Building Your Mobile Strategy7 Factors to Consider When Building Your Mobile Strategy
7 Factors to Consider When Building Your Mobile Strategy
iTexico
 
The Path to 100% Adoption
The Path to 100% AdoptionThe Path to 100% Adoption
The Path to 100% Adoption
dreamforce2006
 

Similar to The Practice of Data Driven Products in Kuaishou (20)

Disruptive Innovation
Disruptive InnovationDisruptive Innovation
Disruptive Innovation
 
Discover Why Less is More in B2B Research
Discover Why Less is More in B2B ResearchDiscover Why Less is More in B2B Research
Discover Why Less is More in B2B Research
 
Intro to Data Analytics with Oscar's Director of Product
 Intro to Data Analytics with Oscar's Director of Product Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product
 
Voice of the Customer (VOC) Research
Voice of the Customer (VOC) ResearchVoice of the Customer (VOC) Research
Voice of the Customer (VOC) Research
 
Purchasing & Value Analysis
Purchasing & Value AnalysisPurchasing & Value Analysis
Purchasing & Value Analysis
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
 
Operations: Top Reasons for Long Lead Times and What to Do About Them
Operations: Top Reasons for Long Lead Times and What to Do About ThemOperations: Top Reasons for Long Lead Times and What to Do About Them
Operations: Top Reasons for Long Lead Times and What to Do About Them
 
How to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your BusinessHow to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your Business
 
La Dove Associates -- CRM/Customer Care Consulting Overview
La Dove Associates --  CRM/Customer Care Consulting Overview La Dove Associates --  CRM/Customer Care Consulting Overview
La Dove Associates -- CRM/Customer Care Consulting Overview
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'ts
 
Dat analytics all verticals
Dat analytics all verticalsDat analytics all verticals
Dat analytics all verticals
 
OKRs KPIs Strategy and Agility.pptx
OKRs KPIs Strategy and Agility.pptxOKRs KPIs Strategy and Agility.pptx
OKRs KPIs Strategy and Agility.pptx
 
ExpOn Conference Presentation - Sao Paulo, Brazil
ExpOn Conference Presentation - Sao Paulo, BrazilExpOn Conference Presentation - Sao Paulo, Brazil
ExpOn Conference Presentation - Sao Paulo, Brazil
 
20/10 Vision: Building A 21st Century Market Research Organization
20/10 Vision: Building A 21st Century Market Research Organization20/10 Vision: Building A 21st Century Market Research Organization
20/10 Vision: Building A 21st Century Market Research Organization
 
Competing on analytics
Competing on analyticsCompeting on analytics
Competing on analytics
 
class
classclass
class
 
Conversion Whitepaper
Conversion WhitepaperConversion Whitepaper
Conversion Whitepaper
 
The Insider's Guide to Workforce Analytics
The Insider's Guide to Workforce AnalyticsThe Insider's Guide to Workforce Analytics
The Insider's Guide to Workforce Analytics
 
7 Factors to Consider When Building Your Mobile Strategy
7 Factors to Consider When Building Your Mobile Strategy7 Factors to Consider When Building Your Mobile Strategy
7 Factors to Consider When Building Your Mobile Strategy
 
The Path to 100% Adoption
The Path to 100% AdoptionThe Path to 100% Adoption
The Path to 100% Adoption
 

More from Jay (Jianqiang) Wang

Artificial Intelligence in fashion -- Combining Statistics and Expert Human J...
Artificial Intelligence in fashion -- Combining Statistics and Expert Human J...Artificial Intelligence in fashion -- Combining Statistics and Expert Human J...
Artificial Intelligence in fashion -- Combining Statistics and Expert Human J...
Jay (Jianqiang) Wang
 
Making data-informed decisions and building intelligent products (Chinese)
Making data-informed decisions and building intelligent products (Chinese)Making data-informed decisions and building intelligent products (Chinese)
Making data-informed decisions and building intelligent products (Chinese)
Jay (Jianqiang) Wang
 
Notes on Machine Learning and Data-centric Startups
Notes on Machine Learning and Data-centric StartupsNotes on Machine Learning and Data-centric Startups
Notes on Machine Learning and Data-centric Startups
Jay (Jianqiang) Wang
 
Boosted multinomial logit model (working manuscript)
Boosted multinomial logit model (working manuscript)Boosted multinomial logit model (working manuscript)
Boosted multinomial logit model (working manuscript)
Jay (Jianqiang) Wang
 
Boosted Tree-based Multinomial Logit Model for Aggregated Market Data
Boosted Tree-based Multinomial Logit Model for Aggregated Market DataBoosted Tree-based Multinomial Logit Model for Aggregated Market Data
Boosted Tree-based Multinomial Logit Model for Aggregated Market Data
Jay (Jianqiang) Wang
 
Multivariate outlier detection
Multivariate outlier detectionMultivariate outlier detection
Multivariate outlier detection
Jay (Jianqiang) Wang
 
Multivariate outlier detection
Multivariate outlier detectionMultivariate outlier detection
Multivariate outlier detection
Jay (Jianqiang) Wang
 
A Bayesian Approach to Estimating Agricultual Yield Based on Multiple Repeat...
A Bayesian Approach to Estimating Agricultual Yield Based on Multiple Repeat...A Bayesian Approach to Estimating Agricultual Yield Based on Multiple Repeat...
A Bayesian Approach to Estimating Agricultual Yield Based on Multiple Repeat...
Jay (Jianqiang) Wang
 

More from Jay (Jianqiang) Wang (8)

Artificial Intelligence in fashion -- Combining Statistics and Expert Human J...
Artificial Intelligence in fashion -- Combining Statistics and Expert Human J...Artificial Intelligence in fashion -- Combining Statistics and Expert Human J...
Artificial Intelligence in fashion -- Combining Statistics and Expert Human J...
 
Making data-informed decisions and building intelligent products (Chinese)
Making data-informed decisions and building intelligent products (Chinese)Making data-informed decisions and building intelligent products (Chinese)
Making data-informed decisions and building intelligent products (Chinese)
 
Notes on Machine Learning and Data-centric Startups
Notes on Machine Learning and Data-centric StartupsNotes on Machine Learning and Data-centric Startups
Notes on Machine Learning and Data-centric Startups
 
Boosted multinomial logit model (working manuscript)
Boosted multinomial logit model (working manuscript)Boosted multinomial logit model (working manuscript)
Boosted multinomial logit model (working manuscript)
 
Boosted Tree-based Multinomial Logit Model for Aggregated Market Data
Boosted Tree-based Multinomial Logit Model for Aggregated Market DataBoosted Tree-based Multinomial Logit Model for Aggregated Market Data
Boosted Tree-based Multinomial Logit Model for Aggregated Market Data
 
Multivariate outlier detection
Multivariate outlier detectionMultivariate outlier detection
Multivariate outlier detection
 
Multivariate outlier detection
Multivariate outlier detectionMultivariate outlier detection
Multivariate outlier detection
 
A Bayesian Approach to Estimating Agricultual Yield Based on Multiple Repeat...
A Bayesian Approach to Estimating Agricultual Yield Based on Multiple Repeat...A Bayesian Approach to Estimating Agricultual Yield Based on Multiple Repeat...
A Bayesian Approach to Estimating Agricultual Yield Based on Multiple Repeat...
 

Recently uploaded

一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
slg6lamcq
 
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
ihavuls
 
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
eoxhsaa
 
Build applications with generative AI on Google Cloud
Build applications with generative AI on Google CloudBuild applications with generative AI on Google Cloud
Build applications with generative AI on Google Cloud
Márton Kodok
 
一比一原版(Sheffield毕业证书)谢菲尔德大学毕业证如何办理
一比一原版(Sheffield毕业证书)谢菲尔德大学毕业证如何办理一比一原版(Sheffield毕业证书)谢菲尔德大学毕业证如何办理
一比一原版(Sheffield毕业证书)谢菲尔德大学毕业证如何办理
1tyxnjpia
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
hyfjgavov
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
Sachin Paul
 
一比一原版加拿大麦吉尔大学毕业证(mcgill毕业证书)如何办理
一比一原版加拿大麦吉尔大学毕业证(mcgill毕业证书)如何办理一比一原版加拿大麦吉尔大学毕业证(mcgill毕业证书)如何办理
一比一原版加拿大麦吉尔大学毕业证(mcgill毕业证书)如何办理
agdhot
 
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
nyvan3
 
Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
ElizabethGarrettChri
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
aqzctr7x
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Kiwi Creative
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
eudsoh
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
Vietnam Cotton & Spinning Association
 
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
uevausa
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
Timothy Spann
 
一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理
keesa2
 
Jio cinema Retention & Engagement Strategy.pdf
Jio cinema Retention & Engagement Strategy.pdfJio cinema Retention & Engagement Strategy.pdf
Jio cinema Retention & Engagement Strategy.pdf
inaya7568
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
Lars Albertsson
 
一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理
zsafxbf
 

Recently uploaded (20)

一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
 
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
 
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
 
Build applications with generative AI on Google Cloud
Build applications with generative AI on Google CloudBuild applications with generative AI on Google Cloud
Build applications with generative AI on Google Cloud
 
一比一原版(Sheffield毕业证书)谢菲尔德大学毕业证如何办理
一比一原版(Sheffield毕业证书)谢菲尔德大学毕业证如何办理一比一原版(Sheffield毕业证书)谢菲尔德大学毕业证如何办理
一比一原版(Sheffield毕业证书)谢菲尔德大学毕业证如何办理
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
 
一比一原版加拿大麦吉尔大学毕业证(mcgill毕业证书)如何办理
一比一原版加拿大麦吉尔大学毕业证(mcgill毕业证书)如何办理一比一原版加拿大麦吉尔大学毕业证(mcgill毕业证书)如何办理
一比一原版加拿大麦吉尔大学毕业证(mcgill毕业证书)如何办理
 
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
 
Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
 
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
 
一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理
 
Jio cinema Retention & Engagement Strategy.pdf
Jio cinema Retention & Engagement Strategy.pdfJio cinema Retention & Engagement Strategy.pdf
Jio cinema Retention & Engagement Strategy.pdf
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
 
一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理
 

The Practice of Data Driven Products in Kuaishou

  • 1. PM-Summit The Practice of Data Driven Products in Kuaishou Jay Wang Former Twitter/Snap Data Scientist Director of Data Science, Kuaishou PM-Summit
  • 2. PM-Summit A g e n d a Data Science Overview Annual plan of a DS team, job responsibility, value 1 Background Knowledge Some pitfalls, analysis methods 2 DS Applications Decision support, metrics, AB testing 3
  • 3. PM-Summit SELF INTRO vData analytics & machine learning vHP Labs: demand forecasting, pricing and portfolio management vTwitter: ads prediction and ranking vStitch fix: e-commerce recommendations vSnap:video ads and creative tools
  • 4. PM-Summit Average DAUs on Kuaishou APP MAU in overseas market daily active streamers Livestreaming user penetration rate Total revenue online marketing services Total e-commerce GMV DAU/MAU pairs of mutual followers From 2021 Q2 earning report
  • 6. PM-Summit IMPACT Ensure the minimum exposure of small and medium-sized live streamers, determine the upper limit of bonus Improve CTR, or reduce complaint rate Scale the analysis by building dashboards, automating the online prediction process Optimize the content review process, propose UTR metric for measuring how video consumption leads to creation
  • 7. PM-Summit CUSTOMER GROWTH ANALYTICS v Increase the supply of customers, make ads auction more competitive v Compared to our competitors, we still have plenty of room to grow daily active customers v Increase number of mid-sized customers to improve the healthiness of ads marketplace v Lay the foundation for long-term revenue growth Background Analytics Impact Dashboard: churning customers Build a scalable solution Customer leads-> review ->sales follow-up Influence a process Overview of active customers Customer life cycle Customer churn analysis and churn prediction Leads generation model
  • 8. PM-Summit A g e n d a Data Science Overview Annual plan of a DS team, job responsibility, value 1 Background Knowledge Some pitfalls, analysis methods 2 DS Applications Decision support, metrics, AB testing 3
  • 9. PM-Summit PITFALLS - SELECTION BIAS v Compare those who saw ads with those who didn’t see ads on their purchases; v Response bias when people volunteer to fill out a questionnaire; v Only measure survivors in a medical experiment; selection bias
  • 10. PM-Summit PITFALLS - SELECTION BIAS selection bias v Add filter reason of“potential_exposure” in serving pipeline, compare sessions with the filter reason in base traffic with users who saw the ad; v Post-stratification v Introduce mortality rate, test the statistical significance of mortality rate between treatment groups How to correct v Compare those who saw ads with those who didn’t see ads on their purchases; v Response bias when people volunteer to fill out a questionnaire; v Only measure survivors in a medical experiment;
  • 11. PM-Summit PITFALLS - SIMPSON’S PARADOX Admission rate by gender Berkeley admission (1974) Work example Admission rate by gender and department Women tend to apply to departments with intense competition and low admission rates, while men tend to apply to departments with less competition and easy to be admitted (such as engineering school, etc.) Search penetration rate drops after summer vacation ends. Might be teenagers are churning from app usage, while teenagers have high search penetration rate Mantel–Haenszel correction Mantel-Haenszel corrected admission rate: men 32%,women 33%
  • 12. PM-Summit PITFALLS - CORRELATION VS CAUSATION Can we conclude that people are more likely to be tangled in their bedsheets because they eat more cheese? Office 365 users that see error messages and experience crashes have lower churn rates Does that mean that Office 365 should show more error messages or that Microsoft should lower code quality, causing more crashes to reduce churn rate? Work example
  • 13. PM-Summit Panel data vs cross-sectional data
  • 14. PM-Summit WHOLE PICTURE OF SEARCH PRODUCT
  • 15. PM-Summit A g e n d a Data Science Overview Annual plan of a DS team, job responsibility, value 1 Background Knowledge Some pitfalls, analysis methods 2 DS Applications Decision support, metrics, AB testing 3
  • 16. PM-Summit DECISION SUPPORT Who What Examples Executives and business stakeholders Define, monitor and report the core metrics of a business line or the company Interpret changes in core business metrics Answer high-level questions that impact decision-making External and internal data, may include competitor and third-party data sources What will be next years’ DAU forecast? Which product lines should the company expand to? What is the ceiling of video search? Open-endedness and uncertainty Trust by top executive leaders Grasp of internal and external information Arguments and quality of report Data Challenges
  • 19. PM-Summit Peter Drucker (“the founder of modern management”) If you cannot measure it, you cannot improve it Goodhart’s Law: When a measure becomes a target, it ceases to be a good measure Campbell's Law: The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor QUOTES ON METRICS Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
  • 20. PM-Summit Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
  • 21. PM-Summit FROM BUSINESS TO EXPERIMENT METRICS • Changes in business • Changes in environment • Changes in understanding of metrics Make sure metrics are not easily “gameable” • e.g. : rats in Seattle Measurable Attributable Sensitive and timely Modify metrics when Metrics and incentives Required by experiment metrics Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
  • 23. PM-Summit ESTABLISHING CAUSALITY Eric Colson (Former Netflix VP) AB testing is the only way to draw causal conclusions. hierarchy of evidence for establishing causality Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
  • 24. PM-Summit 实验平台的成熟度 1. Crawl: Building foundational prerequisites, to compute the summary statistics needed so you can design, run and analyze a few experiments 2. Walk: Focus on defining standard metrics and getting the organization to run more experiments (validating instrumentation, running AA test, SRM tests) 3. Run: Running experiments at scale, with comprehensive metrics and decision policy (agreed-on metrics or OEC metric) 4. Fly: Running experiments as the norm for every change, building institutional memory, feature teams should be able to analyze experiments on their own (kuaishou and bytedance) Reference: trustworthy controlled online experiments by Kohavi, Tang and Xu
  • 25. PM-Summit BE CAUTIONS User learning effect: search button moved from secondary page to front page Independence assumption: sending/receiving messages in social network, experiment in an isolated world Combining multiple experiments: 6 experiments with 1% lift during the quarter, what is the combined effect?
  • 26. PM-Summit A g e n d a Data Science Overview Annual plan of a DS team, job responsibility, value 1 Background Knowledge Some pitfalls, analysis methods 2 DS Applications Decision support, metrics, AB testing 3