SlideShare a Scribd company logo
AI-driven product
innovation
from Recommender Systems to COVID-19
Xavier Amatriain
Co-founder/CTO Curai, former Quora, Netflix, researcher (>5k citations)
May 7, 2021
About me...
● PhD on Audio and Music Signal Processing and Modeling
● Led research on Augmented/Immersive Reality at UCSB
● Researcher in Recommender Systems for several years
● Started and led ML Algorithms at Netflix
● Head of Engineering at Quora
● Currently co-founder/CTO at Curai
2
Outline
1. The recommender problem & the Netflix Prize
2. Recommendations beyond rating prediction
3. AI + Healthcare: A bit about Curai
4. Lessons Learned
1. The
Recommender
Problem
4
The Age of Search has come to an end
● ... long live the Age of Recommendation!
● Chris Anderson in “The Long Tail”
o “We are leaving the age of information and entering the age of
recommendation”
● CNN Money, “The race to create a 'smart' Google”:
o “The Web, they say, is leaving the era of search and entering one of
discovery. What's the difference? Search is what you do when you're
looking for something. Discovery is when something wonderful that
you didn't know existed, or didn't know how to ask for, finds you.”
The “Recommender problem”
● “Traditional” definition: Estimate a utility function that
automatically predicts how much a user will like an
item.
● Based on:
o Past behavior
o Relations to other users
o Item similarity
o Context
o …
Approaches to Recommendation
● Collaborative Filtering: Recommendations based only on user
behavior (target user and similar users’ behavior)
○ Item-based: Find similar items to those that I have liked
○ User-based: Find similar users to me, and recommend what they liked
● Others
○ Content-based: Recommend items similar to the ones I liked based on their
features
○ Demographic: Recommend items liked by users with similar features to me
○ Social: Recommend items liked by people socially connected to me
● Personalized learning-to-rank: Treat recommendations not as
binary like/not like, but rather as a ranking/sorting problem
● Hybrid: Combined any of the above
● Depends on the domain and particular problem
● However, in the general case it has been demonstrated that
the best isolated approach is CF
o Other approaches can be hybridized to improve results in specific cases
(cold-start problem...)
● What matters:
o Data preprocessing: outlier removal, denoising, removal of global effects
(e.g. individual user's average)
o “Smart” dimensionality reduction using MF
o Combining methods through ensembles
What works?
What we were interested in:
▪ High quality recommendations
Proxy question:
▪ Accuracy in predicted rating
▪ Improve by 10% = $1million!
▪ Top 2 algorithms
▪ SVD - Prize RMSE: 0.8914
▪ RBM - Prize RMSE: 0.8990
▪ Linear blend Prize RMSE: 0.88
▪ Limitations
▪ Designed for 100M ratings, not XB ratings
▪ Not adaptable as users add ratings
▪ Performance issues
What about the final prize ensembles?
● Offline studies showed they were too computationally
intensive to scale
● Expected improvement not worth engineering effort
● Plus…. Focus had already shifted to other issues that had more
impact than rating prediction.
Outline
1. The recommender problem & the Netflix Prize
2. Recommendations beyond rating prediction
3. AI + Healthcare: A bit about Curai
4. Lessons Learned
2. Beyond
Rating
Prediction
12
Everything is a recommendation
Evolution of the Recommender Problem
Rating Ranking Page Optimization
4.7
Context-aware
Recommendations
Context
Ranking
● Most recommendations are presented in a sorted list
● Recommendation can be understood as a ranking problem
● Popularity is the obvious baseline, but it is not personalized
● Can we combine it with personalized rating predictions?
Ranking by ratings
4.7 4.6 4.5 4.5 4.5 4.5 4.5 4.5 4.5 4.5
Niche titles
High average ratings… by those who would watch it
RMSE
Popularity
Predicted
Rating
1
2
3
4
5
Linear Model:
frank
(u,v) = w1
p(v) + w2
r(u,v) + b
Final
Ranking
Example: Two features, linear model
Popularity
1
2
3
4
5
Final
Ranking
Predicted
Rating
Example: Two features, linear model
Ranking - Quora Feed
Goal: Present most interesting stories for a
user at a given time
Interesting = topical relevance +
social relevance + timeliness
Stories = questions + answers
ML: Personalized learning-to-rank approach
Relevance-ordered vs time-ordered = big
gains in engagement
10,000s of
possible
rows …
10-40
rows
Variable number of
possible videos per
row (up to thousands)
1 personalized page
per device
Page Composition
User Attention Modeling
From “Modeling User Attention and
Interaction on the Web” 2014 - PhD Thesis by Dmitry Lagun (Emory U.)
N-dimensional model
Deep learning
However...
Outline
1. The recommender problem & the Netflix Prize
2. Recommendations beyond rating prediction
3. AI + Healthcare: A bit about Curai
4. Lessons Learned
3. AI + Healthcare:
A bit about Curai
27
● >50% world with no access
to essential health services
○ ~30% of US adults
under-insured
● ~15 min. to capture
information, diagnose,
recommend treatment
● 30% of the medical errors
causing ~400k deaths a
year are due to
misdiagnosis
Healthcare access, quality, and scalability
shortage of 120,000 physicians by 2030
29
We have an opportunity to
reimagine healthcare
30
We have an opportunity
obligation to reimagine
healthcare
Towards an AI powered learning health system
● Mobile-First Care, always
on, accessible, affordable
● AI + human providers in
the loop for quality care
● Always-Learning system
● AI to operate in-the-wild
(EHR)
FEEDBACK
DATA
MODEL
AI-augmented
medical
conversations
Towards an FDA-approved “AI Doctor”
Clinical vignettes +
automated offline
evaluations
KB Analytics Layer
Security & Privacy
Dataset datasheets
Additional Analytics
A/B Testing
Model lineage
Breakthroughs in AI & healthcare
Research areas at Curai
● Medical Reasoning and Diagnosis
○ Learning from the Experts: combining expert systems and
deep learning
● NLP
○ Medical dialogue summarization
○ Transfer Learning for Similar Questions
○ Medical entity recognition: An active learning approach
● Multimodal healthcare AI
○ Few-shot dermatology image classification
General principles
1. Extensible
a. Data feedback loops
b. Incrementally/iteratively learn from
“physician-in-the-loop”
2. Domain knowledge + Data
3. In the wild
a. Uncertainty in prediction
b. Fall-back to “physician-in-the-loop”
Research at Curai
SOTA Medical reasoning and diagnosis
Our approach to COVID-19
ML + Expert systems for Dx models
female
middle aged
fever
cough
Influenza 16.9
bacterial pneumonia 16.9
acute sinusitis 10.9
asthma 10.9
common cold 10.9
influenza 0.753
bacterial pneumonia 0.205
asthma 0.017
acute sinusitis 0.008
pulmonary tuberculosis 0.007
Inputs
DDx with expert system DDx with ML model
Expert
system
Clinical case
simulator
Clinical cases
DDx
ML
model
Common cold
UTI
Acute bronchitis
Female
Middle-aged
Chronic cough
Nasal congestion
Other data
(e.g. EHR)
COVID-aware modeling
Expert
system
Clinical case
simulator
Clinical cases with
DDx
ML
model
Common cold
UTI
Acute bronchitis
Female
Middle-aged
Chronic cough
Nasal congestion
COVID-19
assessment data
COVID-19
COVID-19
female
middle-age
cough
headache
nose discharge
cigarette smoking
hospital personnel
Examples
Inputs DDx before COVID DDx after COVID
female
middle aged
fever
cough
healthcare worker
influenza
bacterial pneumonia
asthma
COVID-19
influenza
bacterial pneumonia
female
middle aged
fever
cough
nasal congestion
influenza
adenovirus infection
bacterial pneumonia
influenza
COVID-19
adenovirus infection
Question similarity for COVID FAQs
● Transfer learning
● Double-finetuned BERT model
● Handle data sparsity
● Medical domain knowledge through an intermediate QA
binary task
Does A answer Q?
Q A
Pretrained
BERT
BERT
Q1 similar to Q2?
Q1 Q2
BERT
Proceedings of the 2020 ACM SIGKDD
Product in action
44
Medical summarization
Best paper at ACL Workshop on medical conversations 2021
Machine Learning for Healthcare, 2021
EMNLP Findings 2020
Our Approach:
GPT-3-ENS
Hasn't used any thing to help
GPT-3
Priming Inference
GPT-3
GPT-3
Hasn’t used any thing to help other than
hydrocortisone
Used nothing else to help other than
Benadryl and hydrocortisone.
10
Trials
21 Unique Labeled
Examples per
Priming Context
Confidentiality note: In accordance with our privacy policy, the illustrative examples included in this document do NOT correspond to real patients. They are either synthetic or fully anonymized.
Chat Snippet:
DR: Thanks for answering my questions.
Apart from Benadryl and Hydrocortisone,
have you used anything to help?
PT: No that’s everything
GPT-3 vs GPT-3-ENS
● 12.5% of summaries produced by
GPT-3-ENS are better than those of GPT-3
GPT-3-ENS as data synthesizer
Hasn't used any thing to
help
Priming Inference
GPT-3
Hasn’t used any thing to help
other than hydrocortisone
Used nothing else to help
other than Benadryl and
hydrocortisone.
10
Trials
21 Labeled Examples
per Priming Context
GPT-3
GPT-3
GPT-3-ENS Labeled Dataset
+
Doctor Labeled/Corrected Dataset
In-House Summarization Model
● Privacy concerns with
invoking GPT-3 at
inference time
● Doctor in the loop
Confidentiality note: In accordance with our privacy policy, the illustrative examples included in this document do NOT correspond to real patients. They are either synthetic or fully anonymized.
Chat Snippet:
DR: Thanks for ...
PT: No that’s everything
Results
● We are able to train a summarization model using GPT-3-ENS
labeled data (which needed 210 doctor-labeled examples)
comparable in performance to a model trained using 6400
doctor-labeled examples
● We find that a model that is trained on a mix of human and
GPT-3-ENS labeled data does better than a model trained on either
independently
Conclusions
● Healthcare needs to scale quickly, and this has become obvious in a
global pandemic like the one we are facing
● The only way to scale healthcare while improving quality and
accessibility is through technology and SOTA AI
● But, AI cannot be simply “dropped” in the middle of old workflows
and processes
○ It needs to be integrated in end-to-end medical care benefitting both patients and
providers
https://curai.com/work
Outline
1. The recommender problem & the Netflix Prize
2. Recommendations beyond rating prediction
3. AI + Healthcare: A bit about Curai
4. Lessons Learned
4. Lessons
Learned
52
1.Data or and Models?
More data or better models?
Really?
More data or better models?
Sometimes,
it’s not about
more data
More data or better models?
Norvig:
“Google does not have
better Algorithms only
more Data”
Many
features/
low-bias
models
More data or better models?
Sometimes
you might not
need all your
“Big Data”
0 2 4 6 8 10 12 14 16 18 20
Number of Training Examples (in Millions)
Testing
Accuracy
What about Deep Learning?
Year Breakthrough in AI Datasets (First Available) Algorithms (First Proposal)
1994 Human-level spontaneous speech recognition Spoken Wall Street Journal articles and other
texts (1991)
Hidden Markov Model (1984)
1997 IBM Deep Blue defeated Garry Kasparov 700,000 Grandmaster chess games, aka “The
Extended Book” (1991)
Negascout planning algorithm (1983)
2005 Google’s Arabic- and Chinese-to-English
translation
1,8 trillion tokens from Google Web and News
pages (collected in 2005)
Statistical machine translation algorithm (1988)
2011 IBM watson become the world Jeopardy!
Champion
8,6 million documents from Wikipedia,
Wiktionary, Wikiquote, and Project Gutenberg
(updated in 2005)
Mixture-of-Experts algorithm (1991)
2014 Google’s GoogLeNet object classification at
near-human performance
ImageNet corpus of 1,5 million labeled images
and 1,000 object catagories (2010)
Convolution neural network algorithm (1989)
2015 Google’s Deepmind achieved human parity in
playing 29 Atari games by learning general
control from video
Arcade Learning Environment dataset of over
50 Atari games (2013)
Q-learning algorithm (1992)
Average No. Of Years to Breakthrough 3 years 18 years
The average elapsed time between key algorithm proposals and corresponding advances was about 18 years,
whereas the average elapsed time between key dataset availabilities and corresponding advances was less
than 3 years, or about 6 times faster.
Models and
Recipes
Pretrained
Available models trained using OpenNMT
→ English → German
→ German → English
→ English Summarization
→ Multi-way – FR,ES,PT,IT,RO < > FR,ES,PT,IT,RO
More models coming soon:
→ Ubuntu Dialog Dataset
→ Syntactic Parsing
→ Image-to-Text
What about Deep Learning?
More data and better data
2.Implicit user signals
Are Always sometimes better than
Explicit ones
Implicit vs. Explicit
● Many have acknowledged
that implicit feedback is more
useful
● Is implicit feedback really always
more useful?
● If so, why?
● Implicit data is (usually):
○ More dense, and available for all users
○ Better representative of user behavior vs.
user reflection
○ More related to final objective function
○ Better correlated with AB test results
● E.g. Rating vs watching
Implicit vs. Explicit
● However
○ It is not always the case that
direct implicit feedback correlates
well with long-term retention
○ E.g. clickbait
● Solution:
○ Combine different forms of
implicit + explicit to better represent
long-term goal
Implicit vs. Explicit
3. MODEL ACCURACY IS NOT ALL THAT MATTERS
Explanation/Support for Recommendations
Social Support
4. Model training depends not only
on input data
Training a model
● Model will learn according to:
○ Training data (e.g. implicit and explicit)
○ Target function (e.g. probability of user reading an answer)
○ Metric (e.g. precision vs. recall)
● Example 1 (made up):
○ Optimize probability of a user going to the cinema to watch a movie and rate it
“highly” by using purchase history and previous ratings. Use NDCG of the ranking as
final metric using only movies rated 4 or higher as positives.
Example 2 - Quora’s feed
● Training data = implicit + explicit
● Target function: Value of showing a story
to a
user ~ weighted sum of actions:
v = ∑a
va
1{ya
= 1}
○ predict probabilities for each action, then compute expected
value: v_pred = E[ V | x ] = ∑a
va
p(a | x)
● Metric: any ranking metric
5. THERE IS SOMETHING EVEN MORE IMPORTANT THAN
MODELS AND DATA: EXPERIMENTAL PROCESS
Offline/Online testing process
Executing A/B tests
● Measure differences in metrics across statistically identical populations that
each experience a different algorithm.
● Decisions on the product always data-driven
● Overall Evaluation Criteria (OEC) = member retention
○ Use long-term metrics whenever possible
○ Short-term metrics can be informative and allow faster decisions
■ But, not always aligned with OEC
Offline testing
● Measure model performance, using (IR)
metrics
● Offline performance = indication to make
decisions on follow-up A/B tests
● A critical (and mostly unsolved) issue is
how offline metrics correlate with A/B test
results.
5. Conclusions
74
01.
02.
03.
04.
05.
Choose the right metric
Be thoughtful about your data
Understand dependencies between data, models & systems
Optimize only what matters, beware of biases
Be thoughtful about : data infrastructure/tools, how to
organize your teams, and above all, your product
requirements
2. Further
“reading”
76
4 hour lecture on recommendations
Carnegie Mellon (2014)
1 hour lecture on practical Deep Learning
UC Berkeley (2020)
10 minutes on AI for COVID
Stanford (2020)
1 hour podcast on AI for Healthcare
Gradient Dissent (2021)

More Related Content

What's hot

Recommending for the World
Recommending for the WorldRecommending for the World
Recommending for the World
Yves Raimond
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Sudeep Das, Ph.D.
 
Google's Official Note to Product Management Candidates
Google's Official Note to Product Management CandidatesGoogle's Official Note to Product Management Candidates
Google's Official Note to Product Management Candidates
Lewis Lin 🦊
 
WTM-2023-Looker Studio.pdf
WTM-2023-Looker Studio.pdfWTM-2023-Looker Studio.pdf
WTM-2023-Looker Studio.pdf
PyariKumaran
 
Data/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealthData/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealth
Xavier Amatriain
 
Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018 Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018
Fernando Amat
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
Yves Raimond
 
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
MLconf
 
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
Zachary Schendel
 
Big data
Big dataBig data
Data Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8thData Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8th
Jonathan Woodward
 
Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at Netflix
Justin Basilico
 
Visualising Data with Code
Visualising Data with CodeVisualising Data with Code
Visualising Data with Code
Ri Liu
 
Recommending What Video to Watch Next: A Multitask Ranking System
Recommending What Video to Watch Next: A Multitask Ranking SystemRecommending What Video to Watch Next: A Multitask Ranking System
Recommending What Video to Watch Next: A Multitask Ranking System
ivaderivader
 
Machine Learning at Netflix Scale
Machine Learning at Netflix ScaleMachine Learning at Netflix Scale
Machine Learning at Netflix Scale
Aish Fenton
 
Netflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelNetflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time Travel
Faisal Siddiqi
 
Time, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsTime, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender Systems
Yves Raimond
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101
Ashish Dua
 
Is AI generation the next platform shift?
Is AI generation the next platform shift?Is AI generation the next platform shift?
Is AI generation the next platform shift?
Bessemer Venture Partners
 
Homepage Personalization at Spotify
Homepage Personalization at SpotifyHomepage Personalization at Spotify
Homepage Personalization at Spotify
Oguz Semerci
 

What's hot (20)

Recommending for the World
Recommending for the WorldRecommending for the World
Recommending for the World
 
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se... Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
Deeper Things: How Netflix Leverages Deep Learning in Recommendations and Se...
 
Google's Official Note to Product Management Candidates
Google's Official Note to Product Management CandidatesGoogle's Official Note to Product Management Candidates
Google's Official Note to Product Management Candidates
 
WTM-2023-Looker Studio.pdf
WTM-2023-Looker Studio.pdfWTM-2023-Looker Studio.pdf
WTM-2023-Looker Studio.pdf
 
Data/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealthData/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealth
 
Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018 Artwork Personalization at Netflix Fernando Amat RecSys2018
Artwork Personalization at Netflix Fernando Amat RecSys2018
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
 
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
 
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
RecSys 2020 A Human Perspective on Algorithmic Similarity Schendel 9-2020
 
Big data
Big dataBig data
Big data
 
Data Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8thData Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8th
 
Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at Netflix
 
Visualising Data with Code
Visualising Data with CodeVisualising Data with Code
Visualising Data with Code
 
Recommending What Video to Watch Next: A Multitask Ranking System
Recommending What Video to Watch Next: A Multitask Ranking SystemRecommending What Video to Watch Next: A Multitask Ranking System
Recommending What Video to Watch Next: A Multitask Ranking System
 
Machine Learning at Netflix Scale
Machine Learning at Netflix ScaleMachine Learning at Netflix Scale
Machine Learning at Netflix Scale
 
Netflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time TravelNetflix Recommendations Feature Engineering with Time Travel
Netflix Recommendations Feature Engineering with Time Travel
 
Time, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender SystemsTime, Context and Causality in Recommender Systems
Time, Context and Causality in Recommender Systems
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101
 
Is AI generation the next platform shift?
Is AI generation the next platform shift?Is AI generation the next platform shift?
Is AI generation the next platform shift?
 
Homepage Personalization at Spotify
Homepage Personalization at SpotifyHomepage Personalization at Spotify
Homepage Personalization at Spotify
 

Similar to AI-driven product innovation: from Recommender Systems to COVID-19

一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
gcljeuzdu
 
What data scientists really do, according to 50 data scientists
What data scientists really do, according to 50 data scientistsWhat data scientists really do, according to 50 data scientists
What data scientists really do, according to 50 data scientists
Hugo Bowne-Anderson
 
Lessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systemsLessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systems
Xavier Amatriain
 
Recommandation systems -
Recommandation systems - Recommandation systems -
Recommandation systems -
Yousef Fadila
 
Baworld adapting to whats happening
Baworld adapting to whats happeningBaworld adapting to whats happening
Baworld adapting to whats happening
Dave Davis PMP, PgMP, PBA
 
ASA conference Feb 2013
ASA conference Feb 2013ASA conference Feb 2013
ASA conference Feb 2013
mrkwr
 
Step Up Your Survey Research - Dawn of the Data Age Lecture Series
Step Up Your Survey Research - Dawn of the Data Age Lecture SeriesStep Up Your Survey Research - Dawn of the Data Age Lecture Series
Step Up Your Survey Research - Dawn of the Data Age Lecture Series
Luciano Pesci, PhD
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
Xavier Amatriain
 
2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...
Ed Chi
 
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure LeskovecThe Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive
 
Module 3 - Improving Current Business with External Data- Online
Module 3 - Improving Current Business with External Data- Online Module 3 - Improving Current Business with External Data- Online
Module 3 - Improving Current Business with External Data- Online
caniceconsulting
 
Recommender Systems In Industry
Recommender Systems In IndustryRecommender Systems In Industry
Recommender Systems In Industry
Xavier Amatriain
 
Elsevier
ElsevierElsevier
Elsevier
Christina Azzam
 
Understanding Cognitive Applications: A Framework - Sue Feldman
Understanding Cognitive Applications:  A Framework - Sue FeldmanUnderstanding Cognitive Applications:  A Framework - Sue Feldman
Understanding Cognitive Applications: A Framework - Sue Feldman
diannepatricia
 
Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22
Matthias Schuurmans
 

Similar to AI-driven product innovation: from Recommender Systems to COVID-19 (20)

一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
 
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
 
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
 
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
 
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
 
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
 
What data scientists really do, according to 50 data scientists
What data scientists really do, according to 50 data scientistsWhat data scientists really do, according to 50 data scientists
What data scientists really do, according to 50 data scientists
 
Lessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systemsLessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systems
 
Recommandation systems -
Recommandation systems - Recommandation systems -
Recommandation systems -
 
Baworld adapting to whats happening
Baworld adapting to whats happeningBaworld adapting to whats happening
Baworld adapting to whats happening
 
ASA conference Feb 2013
ASA conference Feb 2013ASA conference Feb 2013
ASA conference Feb 2013
 
Step Up Your Survey Research - Dawn of the Data Age Lecture Series
Step Up Your Survey Research - Dawn of the Data Age Lecture SeriesStep Up Your Survey Research - Dawn of the Data Age Lecture Series
Step Up Your Survey Research - Dawn of the Data Age Lecture Series
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
 
2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...
 
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure LeskovecThe Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
 
Module 3 - Improving Current Business with External Data- Online
Module 3 - Improving Current Business with External Data- Online Module 3 - Improving Current Business with External Data- Online
Module 3 - Improving Current Business with External Data- Online
 
Recommender Systems In Industry
Recommender Systems In IndustryRecommender Systems In Industry
Recommender Systems In Industry
 
Elsevier
ElsevierElsevier
Elsevier
 
Understanding Cognitive Applications: A Framework - Sue Feldman
Understanding Cognitive Applications:  A Framework - Sue FeldmanUnderstanding Cognitive Applications:  A Framework - Sue Feldman
Understanding Cognitive Applications: A Framework - Sue Feldman
 
Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22
 

More from Xavier Amatriain

AI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 updateAI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 update
Xavier Amatriain
 
AI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approachAI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approach
Xavier Amatriain
 
AI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for EveryoneAI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for Everyone
Xavier Amatriain
 
Towards online universal quality healthcare through AI
Towards online universal quality healthcare through AITowards online universal quality healthcare through AI
Towards online universal quality healthcare through AI
Xavier Amatriain
 
From one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategyFrom one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategy
Xavier Amatriain
 
Learning to speak medicine
Learning to speak medicineLearning to speak medicine
Learning to speak medicine
Xavier Amatriain
 
ML to cure the world
ML to cure the worldML to cure the world
ML to cure the world
Xavier Amatriain
 
Medical advice as a Recommender System
Medical advice as a Recommender SystemMedical advice as a Recommender System
Medical advice as a Recommender System
Xavier Amatriain
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Xavier Amatriain
 
Past present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry PerspectivePast present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry Perspective
Xavier Amatriain
 
Staying Shallow & Lean in a Deep Learning World
Staying Shallow & Lean in a Deep Learning WorldStaying Shallow & Lean in a Deep Learning World
Staying Shallow & Lean in a Deep Learning World
Xavier Amatriain
 
Machine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora ExampleMachine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora Example
Xavier Amatriain
 
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
Xavier Amatriain
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Xavier Amatriain
 
Barcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons LearnedBarcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons Learned
Xavier Amatriain
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
Xavier Amatriain
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
Xavier Amatriain
 
Machine Learning to Grow the World's Knowledge
Machine Learning to Grow  the World's KnowledgeMachine Learning to Grow  the World's Knowledge
Machine Learning to Grow the World's Knowledge
Xavier Amatriain
 
MLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@QuoraMLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@Quora
Xavier Amatriain
 
Lean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven CompaniesLean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven Companies
Xavier Amatriain
 

More from Xavier Amatriain (20)

AI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 updateAI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 update
 
AI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approachAI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approach
 
AI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for EveryoneAI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for Everyone
 
Towards online universal quality healthcare through AI
Towards online universal quality healthcare through AITowards online universal quality healthcare through AI
Towards online universal quality healthcare through AI
 
From one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategyFrom one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategy
 
Learning to speak medicine
Learning to speak medicineLearning to speak medicine
Learning to speak medicine
 
ML to cure the world
ML to cure the worldML to cure the world
ML to cure the world
 
Medical advice as a Recommender System
Medical advice as a Recommender SystemMedical advice as a Recommender System
Medical advice as a Recommender System
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
 
Past present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry PerspectivePast present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry Perspective
 
Staying Shallow & Lean in a Deep Learning World
Staying Shallow & Lean in a Deep Learning WorldStaying Shallow & Lean in a Deep Learning World
Staying Shallow & Lean in a Deep Learning World
 
Machine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora ExampleMachine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora Example
 
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
 
Barcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons LearnedBarcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons Learned
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
 
Machine Learning to Grow the World's Knowledge
Machine Learning to Grow  the World's KnowledgeMachine Learning to Grow  the World's Knowledge
Machine Learning to Grow the World's Knowledge
 
MLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@QuoraMLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@Quora
 
Lean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven CompaniesLean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven Companies
 

Recently uploaded

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
flufftailshop
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Jeffrey Haguewood
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Intelisync
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
DanBrown980551
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
Pravash Chandra Das
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
alexjohnson7307
 

Recently uploaded (20)

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
 

AI-driven product innovation: from Recommender Systems to COVID-19

  • 1. AI-driven product innovation from Recommender Systems to COVID-19 Xavier Amatriain Co-founder/CTO Curai, former Quora, Netflix, researcher (>5k citations) May 7, 2021
  • 2. About me... ● PhD on Audio and Music Signal Processing and Modeling ● Led research on Augmented/Immersive Reality at UCSB ● Researcher in Recommender Systems for several years ● Started and led ML Algorithms at Netflix ● Head of Engineering at Quora ● Currently co-founder/CTO at Curai 2
  • 3. Outline 1. The recommender problem & the Netflix Prize 2. Recommendations beyond rating prediction 3. AI + Healthcare: A bit about Curai 4. Lessons Learned
  • 5. The Age of Search has come to an end ● ... long live the Age of Recommendation! ● Chris Anderson in “The Long Tail” o “We are leaving the age of information and entering the age of recommendation” ● CNN Money, “The race to create a 'smart' Google”: o “The Web, they say, is leaving the era of search and entering one of discovery. What's the difference? Search is what you do when you're looking for something. Discovery is when something wonderful that you didn't know existed, or didn't know how to ask for, finds you.”
  • 6. The “Recommender problem” ● “Traditional” definition: Estimate a utility function that automatically predicts how much a user will like an item. ● Based on: o Past behavior o Relations to other users o Item similarity o Context o …
  • 7. Approaches to Recommendation ● Collaborative Filtering: Recommendations based only on user behavior (target user and similar users’ behavior) ○ Item-based: Find similar items to those that I have liked ○ User-based: Find similar users to me, and recommend what they liked ● Others ○ Content-based: Recommend items similar to the ones I liked based on their features ○ Demographic: Recommend items liked by users with similar features to me ○ Social: Recommend items liked by people socially connected to me ● Personalized learning-to-rank: Treat recommendations not as binary like/not like, but rather as a ranking/sorting problem ● Hybrid: Combined any of the above
  • 8. ● Depends on the domain and particular problem ● However, in the general case it has been demonstrated that the best isolated approach is CF o Other approaches can be hybridized to improve results in specific cases (cold-start problem...) ● What matters: o Data preprocessing: outlier removal, denoising, removal of global effects (e.g. individual user's average) o “Smart” dimensionality reduction using MF o Combining methods through ensembles What works?
  • 9. What we were interested in: ▪ High quality recommendations Proxy question: ▪ Accuracy in predicted rating ▪ Improve by 10% = $1million! ▪ Top 2 algorithms ▪ SVD - Prize RMSE: 0.8914 ▪ RBM - Prize RMSE: 0.8990 ▪ Linear blend Prize RMSE: 0.88 ▪ Limitations ▪ Designed for 100M ratings, not XB ratings ▪ Not adaptable as users add ratings ▪ Performance issues
  • 10. What about the final prize ensembles? ● Offline studies showed they were too computationally intensive to scale ● Expected improvement not worth engineering effort ● Plus…. Focus had already shifted to other issues that had more impact than rating prediction.
  • 11. Outline 1. The recommender problem & the Netflix Prize 2. Recommendations beyond rating prediction 3. AI + Healthcare: A bit about Curai 4. Lessons Learned
  • 13. Everything is a recommendation
  • 14. Evolution of the Recommender Problem Rating Ranking Page Optimization 4.7 Context-aware Recommendations Context
  • 15. Ranking ● Most recommendations are presented in a sorted list ● Recommendation can be understood as a ranking problem ● Popularity is the obvious baseline, but it is not personalized ● Can we combine it with personalized rating predictions?
  • 16. Ranking by ratings 4.7 4.6 4.5 4.5 4.5 4.5 4.5 4.5 4.5 4.5 Niche titles High average ratings… by those who would watch it RMSE
  • 17. Popularity Predicted Rating 1 2 3 4 5 Linear Model: frank (u,v) = w1 p(v) + w2 r(u,v) + b Final Ranking Example: Two features, linear model
  • 19.
  • 20. Ranking - Quora Feed Goal: Present most interesting stories for a user at a given time Interesting = topical relevance + social relevance + timeliness Stories = questions + answers ML: Personalized learning-to-rank approach Relevance-ordered vs time-ordered = big gains in engagement
  • 21. 10,000s of possible rows … 10-40 rows Variable number of possible videos per row (up to thousands) 1 personalized page per device Page Composition
  • 22. User Attention Modeling From “Modeling User Attention and Interaction on the Web” 2014 - PhD Thesis by Dmitry Lagun (Emory U.)
  • 26. Outline 1. The recommender problem & the Netflix Prize 2. Recommendations beyond rating prediction 3. AI + Healthcare: A bit about Curai 4. Lessons Learned
  • 27. 3. AI + Healthcare: A bit about Curai 27
  • 28. ● >50% world with no access to essential health services ○ ~30% of US adults under-insured ● ~15 min. to capture information, diagnose, recommend treatment ● 30% of the medical errors causing ~400k deaths a year are due to misdiagnosis Healthcare access, quality, and scalability shortage of 120,000 physicians by 2030
  • 29. 29 We have an opportunity to reimagine healthcare
  • 30. 30 We have an opportunity obligation to reimagine healthcare
  • 31. Towards an AI powered learning health system ● Mobile-First Care, always on, accessible, affordable ● AI + human providers in the loop for quality care ● Always-Learning system ● AI to operate in-the-wild (EHR) FEEDBACK DATA MODEL AI-augmented medical conversations
  • 32.
  • 33. Towards an FDA-approved “AI Doctor” Clinical vignettes + automated offline evaluations KB Analytics Layer Security & Privacy Dataset datasheets Additional Analytics A/B Testing Model lineage
  • 34. Breakthroughs in AI & healthcare
  • 35. Research areas at Curai ● Medical Reasoning and Diagnosis ○ Learning from the Experts: combining expert systems and deep learning ● NLP ○ Medical dialogue summarization ○ Transfer Learning for Similar Questions ○ Medical entity recognition: An active learning approach ● Multimodal healthcare AI ○ Few-shot dermatology image classification
  • 36. General principles 1. Extensible a. Data feedback loops b. Incrementally/iteratively learn from “physician-in-the-loop” 2. Domain knowledge + Data 3. In the wild a. Uncertainty in prediction b. Fall-back to “physician-in-the-loop”
  • 38. SOTA Medical reasoning and diagnosis
  • 39. Our approach to COVID-19
  • 40. ML + Expert systems for Dx models female middle aged fever cough Influenza 16.9 bacterial pneumonia 16.9 acute sinusitis 10.9 asthma 10.9 common cold 10.9 influenza 0.753 bacterial pneumonia 0.205 asthma 0.017 acute sinusitis 0.008 pulmonary tuberculosis 0.007 Inputs DDx with expert system DDx with ML model Expert system Clinical case simulator Clinical cases DDx ML model Common cold UTI Acute bronchitis Female Middle-aged Chronic cough Nasal congestion Other data (e.g. EHR)
  • 41. COVID-aware modeling Expert system Clinical case simulator Clinical cases with DDx ML model Common cold UTI Acute bronchitis Female Middle-aged Chronic cough Nasal congestion COVID-19 assessment data COVID-19 COVID-19 female middle-age cough headache nose discharge cigarette smoking hospital personnel
  • 42. Examples Inputs DDx before COVID DDx after COVID female middle aged fever cough healthcare worker influenza bacterial pneumonia asthma COVID-19 influenza bacterial pneumonia female middle aged fever cough nasal congestion influenza adenovirus infection bacterial pneumonia influenza COVID-19 adenovirus infection
  • 43. Question similarity for COVID FAQs ● Transfer learning ● Double-finetuned BERT model ● Handle data sparsity ● Medical domain knowledge through an intermediate QA binary task Does A answer Q? Q A Pretrained BERT BERT Q1 similar to Q2? Q1 Q2 BERT Proceedings of the 2020 ACM SIGKDD
  • 45. Medical summarization Best paper at ACL Workshop on medical conversations 2021 Machine Learning for Healthcare, 2021 EMNLP Findings 2020
  • 46. Our Approach: GPT-3-ENS Hasn't used any thing to help GPT-3 Priming Inference GPT-3 GPT-3 Hasn’t used any thing to help other than hydrocortisone Used nothing else to help other than Benadryl and hydrocortisone. 10 Trials 21 Unique Labeled Examples per Priming Context Confidentiality note: In accordance with our privacy policy, the illustrative examples included in this document do NOT correspond to real patients. They are either synthetic or fully anonymized. Chat Snippet: DR: Thanks for answering my questions. Apart from Benadryl and Hydrocortisone, have you used anything to help? PT: No that’s everything
  • 47. GPT-3 vs GPT-3-ENS ● 12.5% of summaries produced by GPT-3-ENS are better than those of GPT-3
  • 48. GPT-3-ENS as data synthesizer Hasn't used any thing to help Priming Inference GPT-3 Hasn’t used any thing to help other than hydrocortisone Used nothing else to help other than Benadryl and hydrocortisone. 10 Trials 21 Labeled Examples per Priming Context GPT-3 GPT-3 GPT-3-ENS Labeled Dataset + Doctor Labeled/Corrected Dataset In-House Summarization Model ● Privacy concerns with invoking GPT-3 at inference time ● Doctor in the loop Confidentiality note: In accordance with our privacy policy, the illustrative examples included in this document do NOT correspond to real patients. They are either synthetic or fully anonymized. Chat Snippet: DR: Thanks for ... PT: No that’s everything
  • 49. Results ● We are able to train a summarization model using GPT-3-ENS labeled data (which needed 210 doctor-labeled examples) comparable in performance to a model trained using 6400 doctor-labeled examples ● We find that a model that is trained on a mix of human and GPT-3-ENS labeled data does better than a model trained on either independently
  • 50. Conclusions ● Healthcare needs to scale quickly, and this has become obvious in a global pandemic like the one we are facing ● The only way to scale healthcare while improving quality and accessibility is through technology and SOTA AI ● But, AI cannot be simply “dropped” in the middle of old workflows and processes ○ It needs to be integrated in end-to-end medical care benefitting both patients and providers https://curai.com/work
  • 51. Outline 1. The recommender problem & the Netflix Prize 2. Recommendations beyond rating prediction 3. AI + Healthcare: A bit about Curai 4. Lessons Learned
  • 53. 1.Data or and Models?
  • 54. More data or better models? Really?
  • 55. More data or better models? Sometimes, it’s not about more data
  • 56. More data or better models? Norvig: “Google does not have better Algorithms only more Data” Many features/ low-bias models
  • 57. More data or better models? Sometimes you might not need all your “Big Data” 0 2 4 6 8 10 12 14 16 18 20 Number of Training Examples (in Millions) Testing Accuracy
  • 58. What about Deep Learning? Year Breakthrough in AI Datasets (First Available) Algorithms (First Proposal) 1994 Human-level spontaneous speech recognition Spoken Wall Street Journal articles and other texts (1991) Hidden Markov Model (1984) 1997 IBM Deep Blue defeated Garry Kasparov 700,000 Grandmaster chess games, aka “The Extended Book” (1991) Negascout planning algorithm (1983) 2005 Google’s Arabic- and Chinese-to-English translation 1,8 trillion tokens from Google Web and News pages (collected in 2005) Statistical machine translation algorithm (1988) 2011 IBM watson become the world Jeopardy! Champion 8,6 million documents from Wikipedia, Wiktionary, Wikiquote, and Project Gutenberg (updated in 2005) Mixture-of-Experts algorithm (1991) 2014 Google’s GoogLeNet object classification at near-human performance ImageNet corpus of 1,5 million labeled images and 1,000 object catagories (2010) Convolution neural network algorithm (1989) 2015 Google’s Deepmind achieved human parity in playing 29 Atari games by learning general control from video Arcade Learning Environment dataset of over 50 Atari games (2013) Q-learning algorithm (1992) Average No. Of Years to Breakthrough 3 years 18 years The average elapsed time between key algorithm proposals and corresponding advances was about 18 years, whereas the average elapsed time between key dataset availabilities and corresponding advances was less than 3 years, or about 6 times faster.
  • 59. Models and Recipes Pretrained Available models trained using OpenNMT → English → German → German → English → English Summarization → Multi-way – FR,ES,PT,IT,RO < > FR,ES,PT,IT,RO More models coming soon: → Ubuntu Dialog Dataset → Syntactic Parsing → Image-to-Text What about Deep Learning?
  • 60. More data and better data
  • 61. 2.Implicit user signals Are Always sometimes better than Explicit ones
  • 62. Implicit vs. Explicit ● Many have acknowledged that implicit feedback is more useful ● Is implicit feedback really always more useful? ● If so, why?
  • 63. ● Implicit data is (usually): ○ More dense, and available for all users ○ Better representative of user behavior vs. user reflection ○ More related to final objective function ○ Better correlated with AB test results ● E.g. Rating vs watching Implicit vs. Explicit
  • 64. ● However ○ It is not always the case that direct implicit feedback correlates well with long-term retention ○ E.g. clickbait ● Solution: ○ Combine different forms of implicit + explicit to better represent long-term goal Implicit vs. Explicit
  • 65. 3. MODEL ACCURACY IS NOT ALL THAT MATTERS
  • 67. 4. Model training depends not only on input data
  • 68. Training a model ● Model will learn according to: ○ Training data (e.g. implicit and explicit) ○ Target function (e.g. probability of user reading an answer) ○ Metric (e.g. precision vs. recall) ● Example 1 (made up): ○ Optimize probability of a user going to the cinema to watch a movie and rate it “highly” by using purchase history and previous ratings. Use NDCG of the ranking as final metric using only movies rated 4 or higher as positives.
  • 69. Example 2 - Quora’s feed ● Training data = implicit + explicit ● Target function: Value of showing a story to a user ~ weighted sum of actions: v = ∑a va 1{ya = 1} ○ predict probabilities for each action, then compute expected value: v_pred = E[ V | x ] = ∑a va p(a | x) ● Metric: any ranking metric
  • 70. 5. THERE IS SOMETHING EVEN MORE IMPORTANT THAN MODELS AND DATA: EXPERIMENTAL PROCESS
  • 72. Executing A/B tests ● Measure differences in metrics across statistically identical populations that each experience a different algorithm. ● Decisions on the product always data-driven ● Overall Evaluation Criteria (OEC) = member retention ○ Use long-term metrics whenever possible ○ Short-term metrics can be informative and allow faster decisions ■ But, not always aligned with OEC
  • 73. Offline testing ● Measure model performance, using (IR) metrics ● Offline performance = indication to make decisions on follow-up A/B tests ● A critical (and mostly unsolved) issue is how offline metrics correlate with A/B test results.
  • 75. 01. 02. 03. 04. 05. Choose the right metric Be thoughtful about your data Understand dependencies between data, models & systems Optimize only what matters, beware of biases Be thoughtful about : data infrastructure/tools, how to organize your teams, and above all, your product requirements
  • 77. 4 hour lecture on recommendations Carnegie Mellon (2014) 1 hour lecture on practical Deep Learning UC Berkeley (2020) 10 minutes on AI for COVID Stanford (2020) 1 hour podcast on AI for Healthcare Gradient Dissent (2021)