SlideShare a Scribd company logo
AI-driven product
innovation
from Recommender Systems to COVID-19
Xavier Amatriain
Co-founder/CTO Curai, former Quora, Netflix, researcher (>5k citations)
May 7, 2021
About me...
● PhD on Audio and Music Signal Processing and Modeling
● Led research on Augmented/Immersive Reality at UCSB
● Researcher in Recommender Systems for several years
● Started and led ML Algorithms at Netflix
● Head of Engineering at Quora
● Currently co-founder/CTO at Curai
2
Outline
1. The recommender problem & the Netflix Prize
2. Recommendations beyond rating prediction
3. AI + Healthcare: A bit about Curai
4. Lessons Learned
1. The
Recommender
Problem
4
The Age of Search has come to an end
● ... long live the Age of Recommendation!
● Chris Anderson in “The Long Tail”
o “We are leaving the age of information and entering the age of
recommendation”
● CNN Money, “The race to create a 'smart' Google”:
o “The Web, they say, is leaving the era of search and entering one of
discovery. What's the difference? Search is what you do when you're
looking for something. Discovery is when something wonderful that
you didn't know existed, or didn't know how to ask for, finds you.”
The “Recommender problem”
● “Traditional” definition: Estimate a utility function that
automatically predicts how much a user will like an
item.
● Based on:
o Past behavior
o Relations to other users
o Item similarity
o Context
o …
Approaches to Recommendation
● Collaborative Filtering: Recommendations based only on user
behavior (target user and similar users’ behavior)
○ Item-based: Find similar items to those that I have liked
○ User-based: Find similar users to me, and recommend what they liked
● Others
○ Content-based: Recommend items similar to the ones I liked based on their
features
○ Demographic: Recommend items liked by users with similar features to me
○ Social: Recommend items liked by people socially connected to me
● Personalized learning-to-rank: Treat recommendations not as
binary like/not like, but rather as a ranking/sorting problem
● Hybrid: Combined any of the above
● Depends on the domain and particular problem
● However, in the general case it has been demonstrated that
the best isolated approach is CF
o Other approaches can be hybridized to improve results in specific cases
(cold-start problem...)
● What matters:
o Data preprocessing: outlier removal, denoising, removal of global effects
(e.g. individual user's average)
o “Smart” dimensionality reduction using MF
o Combining methods through ensembles
What works?
What we were interested in:
▪ High quality recommendations
Proxy question:
▪ Accuracy in predicted rating
▪ Improve by 10% = $1million!
▪ Top 2 algorithms
▪ SVD - Prize RMSE: 0.8914
▪ RBM - Prize RMSE: 0.8990
▪ Linear blend Prize RMSE: 0.88
▪ Limitations
▪ Designed for 100M ratings, not XB ratings
▪ Not adaptable as users add ratings
▪ Performance issues
What about the final prize ensembles?
● Offline studies showed they were too computationally
intensive to scale
● Expected improvement not worth engineering effort
● Plus…. Focus had already shifted to other issues that had more
impact than rating prediction.
Outline
1. The recommender problem & the Netflix Prize
2. Recommendations beyond rating prediction
3. AI + Healthcare: A bit about Curai
4. Lessons Learned
2. Beyond
Rating
Prediction
12
Everything is a recommendation
Evolution of the Recommender Problem
Rating Ranking Page Optimization
4.7
Context-aware
Recommendations
Context
Ranking
● Most recommendations are presented in a sorted list
● Recommendation can be understood as a ranking problem
● Popularity is the obvious baseline, but it is not personalized
● Can we combine it with personalized rating predictions?
Ranking by ratings
4.7 4.6 4.5 4.5 4.5 4.5 4.5 4.5 4.5 4.5
Niche titles
High average ratings… by those who would watch it
RMSE
Popularity
Predicted
Rating
1
2
3
4
5
Linear Model:
frank
(u,v) = w1
p(v) + w2
r(u,v) + b
Final
Ranking
Example: Two features, linear model
Popularity
1
2
3
4
5
Final
Ranking
Predicted
Rating
Example: Two features, linear model
AI-driven product innovation: from Recommender Systems to COVID-19
Ranking - Quora Feed
Goal: Present most interesting stories for a
user at a given time
Interesting = topical relevance +
social relevance + timeliness
Stories = questions + answers
ML: Personalized learning-to-rank approach
Relevance-ordered vs time-ordered = big
gains in engagement
10,000s of
possible
rows …
10-40
rows
Variable number of
possible videos per
row (up to thousands)
1 personalized page
per device
Page Composition
User Attention Modeling
From “Modeling User Attention and
Interaction on the Web” 2014 - PhD Thesis by Dmitry Lagun (Emory U.)
N-dimensional model
Deep learning
However...
Outline
1. The recommender problem & the Netflix Prize
2. Recommendations beyond rating prediction
3. AI + Healthcare: A bit about Curai
4. Lessons Learned
3. AI + Healthcare:
A bit about Curai
27
● >50% world with no access
to essential health services
○ ~30% of US adults
under-insured
● ~15 min. to capture
information, diagnose,
recommend treatment
● 30% of the medical errors
causing ~400k deaths a
year are due to
misdiagnosis
Healthcare access, quality, and scalability
shortage of 120,000 physicians by 2030
29
We have an opportunity to
reimagine healthcare
30
We have an opportunity
obligation to reimagine
healthcare
Towards an AI powered learning health system
● Mobile-First Care, always
on, accessible, affordable
● AI + human providers in
the loop for quality care
● Always-Learning system
● AI to operate in-the-wild
(EHR)
FEEDBACK
DATA
MODEL
AI-augmented
medical
conversations
AI-driven product innovation: from Recommender Systems to COVID-19
Towards an FDA-approved “AI Doctor”
Clinical vignettes +
automated offline
evaluations
KB Analytics Layer
Security & Privacy
Dataset datasheets
Additional Analytics
A/B Testing
Model lineage
Breakthroughs in AI & healthcare
Research areas at Curai
● Medical Reasoning and Diagnosis
○ Learning from the Experts: combining expert systems and
deep learning
● NLP
○ Medical dialogue summarization
○ Transfer Learning for Similar Questions
○ Medical entity recognition: An active learning approach
● Multimodal healthcare AI
○ Few-shot dermatology image classification
General principles
1. Extensible
a. Data feedback loops
b. Incrementally/iteratively learn from
“physician-in-the-loop”
2. Domain knowledge + Data
3. In the wild
a. Uncertainty in prediction
b. Fall-back to “physician-in-the-loop”
Research at Curai
SOTA Medical reasoning and diagnosis
Our approach to COVID-19
ML + Expert systems for Dx models
female
middle aged
fever
cough
Influenza 16.9
bacterial pneumonia 16.9
acute sinusitis 10.9
asthma 10.9
common cold 10.9
influenza 0.753
bacterial pneumonia 0.205
asthma 0.017
acute sinusitis 0.008
pulmonary tuberculosis 0.007
Inputs
DDx with expert system DDx with ML model
Expert
system
Clinical case
simulator
Clinical cases
DDx
ML
model
Common cold
UTI
Acute bronchitis
Female
Middle-aged
Chronic cough
Nasal congestion
Other data
(e.g. EHR)
COVID-aware modeling
Expert
system
Clinical case
simulator
Clinical cases with
DDx
ML
model
Common cold
UTI
Acute bronchitis
Female
Middle-aged
Chronic cough
Nasal congestion
COVID-19
assessment data
COVID-19
COVID-19
female
middle-age
cough
headache
nose discharge
cigarette smoking
hospital personnel
Examples
Inputs DDx before COVID DDx after COVID
female
middle aged
fever
cough
healthcare worker
influenza
bacterial pneumonia
asthma
COVID-19
influenza
bacterial pneumonia
female
middle aged
fever
cough
nasal congestion
influenza
adenovirus infection
bacterial pneumonia
influenza
COVID-19
adenovirus infection
Question similarity for COVID FAQs
● Transfer learning
● Double-finetuned BERT model
● Handle data sparsity
● Medical domain knowledge through an intermediate QA
binary task
Does A answer Q?
Q A
Pretrained
BERT
BERT
Q1 similar to Q2?
Q1 Q2
BERT
Proceedings of the 2020 ACM SIGKDD
Product in action
44
Medical summarization
Best paper at ACL Workshop on medical conversations 2021
Machine Learning for Healthcare, 2021
EMNLP Findings 2020
Our Approach:
GPT-3-ENS
Hasn't used any thing to help
GPT-3
Priming Inference
GPT-3
GPT-3
Hasn’t used any thing to help other than
hydrocortisone
Used nothing else to help other than
Benadryl and hydrocortisone.
10
Trials
21 Unique Labeled
Examples per
Priming Context
Confidentiality note: In accordance with our privacy policy, the illustrative examples included in this document do NOT correspond to real patients. They are either synthetic or fully anonymized.
Chat Snippet:
DR: Thanks for answering my questions.
Apart from Benadryl and Hydrocortisone,
have you used anything to help?
PT: No that’s everything
GPT-3 vs GPT-3-ENS
● 12.5% of summaries produced by
GPT-3-ENS are better than those of GPT-3
GPT-3-ENS as data synthesizer
Hasn't used any thing to
help
Priming Inference
GPT-3
Hasn’t used any thing to help
other than hydrocortisone
Used nothing else to help
other than Benadryl and
hydrocortisone.
10
Trials
21 Labeled Examples
per Priming Context
GPT-3
GPT-3
GPT-3-ENS Labeled Dataset
+
Doctor Labeled/Corrected Dataset
In-House Summarization Model
● Privacy concerns with
invoking GPT-3 at
inference time
● Doctor in the loop
Confidentiality note: In accordance with our privacy policy, the illustrative examples included in this document do NOT correspond to real patients. They are either synthetic or fully anonymized.
Chat Snippet:
DR: Thanks for ...
PT: No that’s everything
Results
● We are able to train a summarization model using GPT-3-ENS
labeled data (which needed 210 doctor-labeled examples)
comparable in performance to a model trained using 6400
doctor-labeled examples
● We find that a model that is trained on a mix of human and
GPT-3-ENS labeled data does better than a model trained on either
independently
Conclusions
● Healthcare needs to scale quickly, and this has become obvious in a
global pandemic like the one we are facing
● The only way to scale healthcare while improving quality and
accessibility is through technology and SOTA AI
● But, AI cannot be simply “dropped” in the middle of old workflows
and processes
○ It needs to be integrated in end-to-end medical care benefitting both patients and
providers
https://curai.com/work
Outline
1. The recommender problem & the Netflix Prize
2. Recommendations beyond rating prediction
3. AI + Healthcare: A bit about Curai
4. Lessons Learned
4. Lessons
Learned
52
1.Data or and Models?
More data or better models?
Really?
More data or better models?
Sometimes,
it’s not about
more data
More data or better models?
Norvig:
“Google does not have
better Algorithms only
more Data”
Many
features/
low-bias
models
More data or better models?
Sometimes
you might not
need all your
“Big Data”
0 2 4 6 8 10 12 14 16 18 20
Number of Training Examples (in Millions)
Testing
Accuracy
What about Deep Learning?
Year Breakthrough in AI Datasets (First Available) Algorithms (First Proposal)
1994 Human-level spontaneous speech recognition Spoken Wall Street Journal articles and other
texts (1991)
Hidden Markov Model (1984)
1997 IBM Deep Blue defeated Garry Kasparov 700,000 Grandmaster chess games, aka “The
Extended Book” (1991)
Negascout planning algorithm (1983)
2005 Google’s Arabic- and Chinese-to-English
translation
1,8 trillion tokens from Google Web and News
pages (collected in 2005)
Statistical machine translation algorithm (1988)
2011 IBM watson become the world Jeopardy!
Champion
8,6 million documents from Wikipedia,
Wiktionary, Wikiquote, and Project Gutenberg
(updated in 2005)
Mixture-of-Experts algorithm (1991)
2014 Google’s GoogLeNet object classification at
near-human performance
ImageNet corpus of 1,5 million labeled images
and 1,000 object catagories (2010)
Convolution neural network algorithm (1989)
2015 Google’s Deepmind achieved human parity in
playing 29 Atari games by learning general
control from video
Arcade Learning Environment dataset of over
50 Atari games (2013)
Q-learning algorithm (1992)
Average No. Of Years to Breakthrough 3 years 18 years
The average elapsed time between key algorithm proposals and corresponding advances was about 18 years,
whereas the average elapsed time between key dataset availabilities and corresponding advances was less
than 3 years, or about 6 times faster.
Models and
Recipes
Pretrained
Available models trained using OpenNMT
→ English → German
→ German → English
→ English Summarization
→ Multi-way – FR,ES,PT,IT,RO < > FR,ES,PT,IT,RO
More models coming soon:
→ Ubuntu Dialog Dataset
→ Syntactic Parsing
→ Image-to-Text
What about Deep Learning?
More data and better data
2.Implicit user signals
Are Always sometimes better than
Explicit ones
Implicit vs. Explicit
● Many have acknowledged
that implicit feedback is more
useful
● Is implicit feedback really always
more useful?
● If so, why?
● Implicit data is (usually):
○ More dense, and available for all users
○ Better representative of user behavior vs.
user reflection
○ More related to final objective function
○ Better correlated with AB test results
● E.g. Rating vs watching
Implicit vs. Explicit
● However
○ It is not always the case that
direct implicit feedback correlates
well with long-term retention
○ E.g. clickbait
● Solution:
○ Combine different forms of
implicit + explicit to better represent
long-term goal
Implicit vs. Explicit
3. MODEL ACCURACY IS NOT ALL THAT MATTERS
Explanation/Support for Recommendations
Social Support
4. Model training depends not only
on input data
Training a model
● Model will learn according to:
○ Training data (e.g. implicit and explicit)
○ Target function (e.g. probability of user reading an answer)
○ Metric (e.g. precision vs. recall)
● Example 1 (made up):
○ Optimize probability of a user going to the cinema to watch a movie and rate it
“highly” by using purchase history and previous ratings. Use NDCG of the ranking as
final metric using only movies rated 4 or higher as positives.
Example 2 - Quora’s feed
● Training data = implicit + explicit
● Target function: Value of showing a story
to a
user ~ weighted sum of actions:
v = ∑a
va
1{ya
= 1}
○ predict probabilities for each action, then compute expected
value: v_pred = E[ V | x ] = ∑a
va
p(a | x)
● Metric: any ranking metric
5. THERE IS SOMETHING EVEN MORE IMPORTANT THAN
MODELS AND DATA: EXPERIMENTAL PROCESS
Offline/Online testing process
Executing A/B tests
● Measure differences in metrics across statistically identical populations that
each experience a different algorithm.
● Decisions on the product always data-driven
● Overall Evaluation Criteria (OEC) = member retention
○ Use long-term metrics whenever possible
○ Short-term metrics can be informative and allow faster decisions
■ But, not always aligned with OEC
Offline testing
● Measure model performance, using (IR)
metrics
● Offline performance = indication to make
decisions on follow-up A/B tests
● A critical (and mostly unsolved) issue is
how offline metrics correlate with A/B test
results.
5. Conclusions
74
01.
02.
03.
04.
05.
Choose the right metric
Be thoughtful about your data
Understand dependencies between data, models & systems
Optimize only what matters, beware of biases
Be thoughtful about : data infrastructure/tools, how to
organize your teams, and above all, your product
requirements
2. Further
“reading”
76
4 hour lecture on recommendations
Carnegie Mellon (2014)
1 hour lecture on practical Deep Learning
UC Berkeley (2020)
10 minutes on AI for COVID
Stanford (2020)
1 hour podcast on AI for Healthcare
Gradient Dissent (2021)

More Related Content

What's hot

The Chatbots Are Coming: A Guide to Chatbots, AI and Conversational Interfaces
The Chatbots Are Coming: A Guide to Chatbots, AI and Conversational InterfacesThe Chatbots Are Coming: A Guide to Chatbots, AI and Conversational Interfaces
The Chatbots Are Coming: A Guide to Chatbots, AI and Conversational Interfaces
TWG
 
Chatbot
ChatbotChatbot
Chat Bots Presentation 8.9.16
Chat Bots Presentation 8.9.16Chat Bots Presentation 8.9.16
Chat Bots Presentation 8.9.16
Samuel Adams, MBA
 
Large Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdfLarge Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdf
David Rostcheck
 
AI and Accountability
AI and AccountabilityAI and Accountability
AI and Accountability
Hiroshi Nakagawa
 
The Five Levels of Generative AI for Games
The Five Levels of Generative AI for GamesThe Five Levels of Generative AI for Games
The Five Levels of Generative AI for Games
Jon Radoff
 
ChatGPT - AI.pdf
ChatGPT - AI.pdfChatGPT - AI.pdf
ChatGPT - AI.pdf
Bannoon1
 
Tweet sentiment analysis
Tweet sentiment analysisTweet sentiment analysis
Tweet sentiment analysis
Anil Shrestha
 
Demystifying Artificial Intelligence
Demystifying Artificial IntelligenceDemystifying Artificial Intelligence
Demystifying Artificial Intelligence
Scopernia
 
The updated non-technical introduction to ChatGPT SEDA March 2023.pptx
The updated non-technical introduction to ChatGPT SEDA March 2023.pptxThe updated non-technical introduction to ChatGPT SEDA March 2023.pptx
The updated non-technical introduction to ChatGPT SEDA March 2023.pptx
Sue Beckingham
 
What is ChatGPT
What is ChatGPTWhat is ChatGPT
What is ChatGPT
jeetendra mandal
 
Let's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchersLet's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchers
Steven Van Vaerenbergh
 
Intro to LLMs
Intro to LLMsIntro to LLMs
Intro to LLMs
Loic Merckel
 
TEDx Manchester: AI & The Future of Work
TEDx Manchester: AI & The Future of WorkTEDx Manchester: AI & The Future of Work
TEDx Manchester: AI & The Future of Work
Volker Hirsch
 
Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AI
Lora Aroyo
 
The World of ChatGPT.pdf
The World of ChatGPT.pdfThe World of ChatGPT.pdf
The World of ChatGPT.pdf
Ria Morgan
 
Issues on Artificial Intelligence and Future (Standards Perspective)
Issues on Artificial Intelligence  and Future (Standards Perspective)Issues on Artificial Intelligence  and Future (Standards Perspective)
Issues on Artificial Intelligence and Future (Standards Perspective)
Seungyun Lee
 
Sentiment Analysis
Sentiment AnalysisSentiment Analysis
Sentiment Analysis
Aditya Nag
 
Unlocking the Power of ChatGPT and AI in Testing - NextSteps, presented by Ap...
Unlocking the Power of ChatGPT and AI in Testing - NextSteps, presented by Ap...Unlocking the Power of ChatGPT and AI in Testing - NextSteps, presented by Ap...
Unlocking the Power of ChatGPT and AI in Testing - NextSteps, presented by Ap...
Applitools
 
The 7 Biggest Artificial Intelligence (AI) Trends In 2022
The 7 Biggest Artificial Intelligence (AI) Trends In 2022The 7 Biggest Artificial Intelligence (AI) Trends In 2022
The 7 Biggest Artificial Intelligence (AI) Trends In 2022
Bernard Marr
 

What's hot (20)

The Chatbots Are Coming: A Guide to Chatbots, AI and Conversational Interfaces
The Chatbots Are Coming: A Guide to Chatbots, AI and Conversational InterfacesThe Chatbots Are Coming: A Guide to Chatbots, AI and Conversational Interfaces
The Chatbots Are Coming: A Guide to Chatbots, AI and Conversational Interfaces
 
Chatbot
ChatbotChatbot
Chatbot
 
Chat Bots Presentation 8.9.16
Chat Bots Presentation 8.9.16Chat Bots Presentation 8.9.16
Chat Bots Presentation 8.9.16
 
Large Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdfLarge Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdf
 
AI and Accountability
AI and AccountabilityAI and Accountability
AI and Accountability
 
The Five Levels of Generative AI for Games
The Five Levels of Generative AI for GamesThe Five Levels of Generative AI for Games
The Five Levels of Generative AI for Games
 
ChatGPT - AI.pdf
ChatGPT - AI.pdfChatGPT - AI.pdf
ChatGPT - AI.pdf
 
Tweet sentiment analysis
Tweet sentiment analysisTweet sentiment analysis
Tweet sentiment analysis
 
Demystifying Artificial Intelligence
Demystifying Artificial IntelligenceDemystifying Artificial Intelligence
Demystifying Artificial Intelligence
 
The updated non-technical introduction to ChatGPT SEDA March 2023.pptx
The updated non-technical introduction to ChatGPT SEDA March 2023.pptxThe updated non-technical introduction to ChatGPT SEDA March 2023.pptx
The updated non-technical introduction to ChatGPT SEDA March 2023.pptx
 
What is ChatGPT
What is ChatGPTWhat is ChatGPT
What is ChatGPT
 
Let's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchersLet's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchers
 
Intro to LLMs
Intro to LLMsIntro to LLMs
Intro to LLMs
 
TEDx Manchester: AI & The Future of Work
TEDx Manchester: AI & The Future of WorkTEDx Manchester: AI & The Future of Work
TEDx Manchester: AI & The Future of Work
 
Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AI
 
The World of ChatGPT.pdf
The World of ChatGPT.pdfThe World of ChatGPT.pdf
The World of ChatGPT.pdf
 
Issues on Artificial Intelligence and Future (Standards Perspective)
Issues on Artificial Intelligence  and Future (Standards Perspective)Issues on Artificial Intelligence  and Future (Standards Perspective)
Issues on Artificial Intelligence and Future (Standards Perspective)
 
Sentiment Analysis
Sentiment AnalysisSentiment Analysis
Sentiment Analysis
 
Unlocking the Power of ChatGPT and AI in Testing - NextSteps, presented by Ap...
Unlocking the Power of ChatGPT and AI in Testing - NextSteps, presented by Ap...Unlocking the Power of ChatGPT and AI in Testing - NextSteps, presented by Ap...
Unlocking the Power of ChatGPT and AI in Testing - NextSteps, presented by Ap...
 
The 7 Biggest Artificial Intelligence (AI) Trends In 2022
The 7 Biggest Artificial Intelligence (AI) Trends In 2022The 7 Biggest Artificial Intelligence (AI) Trends In 2022
The 7 Biggest Artificial Intelligence (AI) Trends In 2022
 

Similar to AI-driven product innovation: from Recommender Systems to COVID-19

Data/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealthData/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealth
Xavier Amatriain
 
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
gcljeuzdu
 
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
gcljeuzdu
 
What data scientists really do, according to 50 data scientists
What data scientists really do, according to 50 data scientistsWhat data scientists really do, according to 50 data scientists
What data scientists really do, according to 50 data scientists
Hugo Bowne-Anderson
 
Lessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systemsLessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systems
Xavier Amatriain
 
Recommandation systems -
Recommandation systems - Recommandation systems -
Recommandation systems -
Yousef Fadila
 
Baworld adapting to whats happening
Baworld adapting to whats happeningBaworld adapting to whats happening
Baworld adapting to whats happening
Dave Davis PMP, PgMP, PBA
 
ASA conference Feb 2013
ASA conference Feb 2013ASA conference Feb 2013
ASA conference Feb 2013
mrkwr
 
Step Up Your Survey Research - Dawn of the Data Age Lecture Series
Step Up Your Survey Research - Dawn of the Data Age Lecture SeriesStep Up Your Survey Research - Dawn of the Data Age Lecture Series
Step Up Your Survey Research - Dawn of the Data Age Lecture Series
Luciano Pesci, PhD
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
Xavier Amatriain
 
2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...
Ed Chi
 
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure LeskovecThe Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive
 
Module 3 - Improving Current Business with External Data- Online
Module 3 - Improving Current Business with External Data- Online Module 3 - Improving Current Business with External Data- Online
Module 3 - Improving Current Business with External Data- Online
caniceconsulting
 
Recommender Systems In Industry
Recommender Systems In IndustryRecommender Systems In Industry
Recommender Systems In Industry
Xavier Amatriain
 
Elsevier
ElsevierElsevier
Elsevier
Christina Azzam
 
Understanding Cognitive Applications: A Framework - Sue Feldman
Understanding Cognitive Applications:  A Framework - Sue FeldmanUnderstanding Cognitive Applications:  A Framework - Sue Feldman
Understanding Cognitive Applications: A Framework - Sue Feldman
diannepatricia
 

Similar to AI-driven product innovation: from Recommender Systems to COVID-19 (20)

Data/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealthData/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealth
 
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
一比一原版芝加哥大学毕业证(Chicago毕业证)成绩单留信认证
 
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
一比一原版哥伦比亚大学毕业证(Columbia毕业证)成绩单留信认证
 
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
一比一原版加州大学伯克利分校毕业证(UCB毕业证)成绩单留信认证
 
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
一比一原版埃默里大学毕业证(Emory毕业证)成绩单留信认证
 
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
一比一原版塔夫斯大学毕业证(Tufts毕业证)成绩单留信认证
 
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
一比一原版莱斯大学毕业证(Rice毕业证)成绩单留信认证
 
What data scientists really do, according to 50 data scientists
What data scientists really do, according to 50 data scientistsWhat data scientists really do, according to 50 data scientists
What data scientists really do, according to 50 data scientists
 
Lessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systemsLessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systems
 
Recommandation systems -
Recommandation systems - Recommandation systems -
Recommandation systems -
 
Baworld adapting to whats happening
Baworld adapting to whats happeningBaworld adapting to whats happening
Baworld adapting to whats happening
 
ASA conference Feb 2013
ASA conference Feb 2013ASA conference Feb 2013
ASA conference Feb 2013
 
Step Up Your Survey Research - Dawn of the Data Age Lecture Series
Step Up Your Survey Research - Dawn of the Data Age Lecture SeriesStep Up Your Survey Research - Dawn of the Data Age Lecture Series
Step Up Your Survey Research - Dawn of the Data Age Lecture Series
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
 
2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...
 
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure LeskovecThe Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
The Hive Think Tank: Machine Learning at Pinterest by Jure Leskovec
 
Module 3 - Improving Current Business with External Data- Online
Module 3 - Improving Current Business with External Data- Online Module 3 - Improving Current Business with External Data- Online
Module 3 - Improving Current Business with External Data- Online
 
Recommender Systems In Industry
Recommender Systems In IndustryRecommender Systems In Industry
Recommender Systems In Industry
 
Elsevier
ElsevierElsevier
Elsevier
 
Understanding Cognitive Applications: A Framework - Sue Feldman
Understanding Cognitive Applications:  A Framework - Sue FeldmanUnderstanding Cognitive Applications:  A Framework - Sue Feldman
Understanding Cognitive Applications: A Framework - Sue Feldman
 

More from Xavier Amatriain

AI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 updateAI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 update
Xavier Amatriain
 
AI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approachAI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approach
Xavier Amatriain
 
AI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for EveryoneAI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for Everyone
Xavier Amatriain
 
Towards online universal quality healthcare through AI
Towards online universal quality healthcare through AITowards online universal quality healthcare through AI
Towards online universal quality healthcare through AI
Xavier Amatriain
 
From one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategyFrom one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategy
Xavier Amatriain
 
Learning to speak medicine
Learning to speak medicineLearning to speak medicine
Learning to speak medicine
Xavier Amatriain
 
ML to cure the world
ML to cure the worldML to cure the world
ML to cure the world
Xavier Amatriain
 
Medical advice as a Recommender System
Medical advice as a Recommender SystemMedical advice as a Recommender System
Medical advice as a Recommender System
Xavier Amatriain
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Xavier Amatriain
 
Past present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry PerspectivePast present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry Perspective
Xavier Amatriain
 
Staying Shallow & Lean in a Deep Learning World
Staying Shallow & Lean in a Deep Learning WorldStaying Shallow & Lean in a Deep Learning World
Staying Shallow & Lean in a Deep Learning World
Xavier Amatriain
 
Machine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora ExampleMachine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora Example
Xavier Amatriain
 
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
Xavier Amatriain
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Xavier Amatriain
 
Barcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons LearnedBarcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons Learned
Xavier Amatriain
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
Xavier Amatriain
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
Xavier Amatriain
 
Machine Learning to Grow the World's Knowledge
Machine Learning to Grow  the World's KnowledgeMachine Learning to Grow  the World's Knowledge
Machine Learning to Grow the World's Knowledge
Xavier Amatriain
 
MLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@QuoraMLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@Quora
Xavier Amatriain
 
Lean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven CompaniesLean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven Companies
Xavier Amatriain
 

More from Xavier Amatriain (20)

AI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 updateAI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 update
 
AI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approachAI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approach
 
AI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for EveryoneAI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for Everyone
 
Towards online universal quality healthcare through AI
Towards online universal quality healthcare through AITowards online universal quality healthcare through AI
Towards online universal quality healthcare through AI
 
From one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategyFrom one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategy
 
Learning to speak medicine
Learning to speak medicineLearning to speak medicine
Learning to speak medicine
 
ML to cure the world
ML to cure the worldML to cure the world
ML to cure the world
 
Medical advice as a Recommender System
Medical advice as a Recommender SystemMedical advice as a Recommender System
Medical advice as a Recommender System
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
 
Past present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry PerspectivePast present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry Perspective
 
Staying Shallow & Lean in a Deep Learning World
Staying Shallow & Lean in a Deep Learning WorldStaying Shallow & Lean in a Deep Learning World
Staying Shallow & Lean in a Deep Learning World
 
Machine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora ExampleMachine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora Example
 
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
 
Barcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons LearnedBarcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons Learned
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
 
Machine Learning to Grow the World's Knowledge
Machine Learning to Grow  the World's KnowledgeMachine Learning to Grow  the World's Knowledge
Machine Learning to Grow the World's Knowledge
 
MLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@QuoraMLConf Seattle 2015 - ML@Quora
MLConf Seattle 2015 - ML@Quora
 
Lean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven CompaniesLean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven Companies
 

Recently uploaded

Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
kumarjarun2010
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
BrainSell Technologies
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
Anant Gupta
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Zilliz
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
LINUS PROJECTS (INDIA)
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 

Recently uploaded (20)

Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 

AI-driven product innovation: from Recommender Systems to COVID-19

  • 1. AI-driven product innovation from Recommender Systems to COVID-19 Xavier Amatriain Co-founder/CTO Curai, former Quora, Netflix, researcher (>5k citations) May 7, 2021
  • 2. About me... ● PhD on Audio and Music Signal Processing and Modeling ● Led research on Augmented/Immersive Reality at UCSB ● Researcher in Recommender Systems for several years ● Started and led ML Algorithms at Netflix ● Head of Engineering at Quora ● Currently co-founder/CTO at Curai 2
  • 3. Outline 1. The recommender problem & the Netflix Prize 2. Recommendations beyond rating prediction 3. AI + Healthcare: A bit about Curai 4. Lessons Learned
  • 5. The Age of Search has come to an end ● ... long live the Age of Recommendation! ● Chris Anderson in “The Long Tail” o “We are leaving the age of information and entering the age of recommendation” ● CNN Money, “The race to create a 'smart' Google”: o “The Web, they say, is leaving the era of search and entering one of discovery. What's the difference? Search is what you do when you're looking for something. Discovery is when something wonderful that you didn't know existed, or didn't know how to ask for, finds you.”
  • 6. The “Recommender problem” ● “Traditional” definition: Estimate a utility function that automatically predicts how much a user will like an item. ● Based on: o Past behavior o Relations to other users o Item similarity o Context o …
  • 7. Approaches to Recommendation ● Collaborative Filtering: Recommendations based only on user behavior (target user and similar users’ behavior) ○ Item-based: Find similar items to those that I have liked ○ User-based: Find similar users to me, and recommend what they liked ● Others ○ Content-based: Recommend items similar to the ones I liked based on their features ○ Demographic: Recommend items liked by users with similar features to me ○ Social: Recommend items liked by people socially connected to me ● Personalized learning-to-rank: Treat recommendations not as binary like/not like, but rather as a ranking/sorting problem ● Hybrid: Combined any of the above
  • 8. ● Depends on the domain and particular problem ● However, in the general case it has been demonstrated that the best isolated approach is CF o Other approaches can be hybridized to improve results in specific cases (cold-start problem...) ● What matters: o Data preprocessing: outlier removal, denoising, removal of global effects (e.g. individual user's average) o “Smart” dimensionality reduction using MF o Combining methods through ensembles What works?
  • 9. What we were interested in: ▪ High quality recommendations Proxy question: ▪ Accuracy in predicted rating ▪ Improve by 10% = $1million! ▪ Top 2 algorithms ▪ SVD - Prize RMSE: 0.8914 ▪ RBM - Prize RMSE: 0.8990 ▪ Linear blend Prize RMSE: 0.88 ▪ Limitations ▪ Designed for 100M ratings, not XB ratings ▪ Not adaptable as users add ratings ▪ Performance issues
  • 10. What about the final prize ensembles? ● Offline studies showed they were too computationally intensive to scale ● Expected improvement not worth engineering effort ● Plus…. Focus had already shifted to other issues that had more impact than rating prediction.
  • 11. Outline 1. The recommender problem & the Netflix Prize 2. Recommendations beyond rating prediction 3. AI + Healthcare: A bit about Curai 4. Lessons Learned
  • 13. Everything is a recommendation
  • 14. Evolution of the Recommender Problem Rating Ranking Page Optimization 4.7 Context-aware Recommendations Context
  • 15. Ranking ● Most recommendations are presented in a sorted list ● Recommendation can be understood as a ranking problem ● Popularity is the obvious baseline, but it is not personalized ● Can we combine it with personalized rating predictions?
  • 16. Ranking by ratings 4.7 4.6 4.5 4.5 4.5 4.5 4.5 4.5 4.5 4.5 Niche titles High average ratings… by those who would watch it RMSE
  • 17. Popularity Predicted Rating 1 2 3 4 5 Linear Model: frank (u,v) = w1 p(v) + w2 r(u,v) + b Final Ranking Example: Two features, linear model
  • 20. Ranking - Quora Feed Goal: Present most interesting stories for a user at a given time Interesting = topical relevance + social relevance + timeliness Stories = questions + answers ML: Personalized learning-to-rank approach Relevance-ordered vs time-ordered = big gains in engagement
  • 21. 10,000s of possible rows … 10-40 rows Variable number of possible videos per row (up to thousands) 1 personalized page per device Page Composition
  • 22. User Attention Modeling From “Modeling User Attention and Interaction on the Web” 2014 - PhD Thesis by Dmitry Lagun (Emory U.)
  • 26. Outline 1. The recommender problem & the Netflix Prize 2. Recommendations beyond rating prediction 3. AI + Healthcare: A bit about Curai 4. Lessons Learned
  • 27. 3. AI + Healthcare: A bit about Curai 27
  • 28. ● >50% world with no access to essential health services ○ ~30% of US adults under-insured ● ~15 min. to capture information, diagnose, recommend treatment ● 30% of the medical errors causing ~400k deaths a year are due to misdiagnosis Healthcare access, quality, and scalability shortage of 120,000 physicians by 2030
  • 29. 29 We have an opportunity to reimagine healthcare
  • 30. 30 We have an opportunity obligation to reimagine healthcare
  • 31. Towards an AI powered learning health system ● Mobile-First Care, always on, accessible, affordable ● AI + human providers in the loop for quality care ● Always-Learning system ● AI to operate in-the-wild (EHR) FEEDBACK DATA MODEL AI-augmented medical conversations
  • 33. Towards an FDA-approved “AI Doctor” Clinical vignettes + automated offline evaluations KB Analytics Layer Security & Privacy Dataset datasheets Additional Analytics A/B Testing Model lineage
  • 34. Breakthroughs in AI & healthcare
  • 35. Research areas at Curai ● Medical Reasoning and Diagnosis ○ Learning from the Experts: combining expert systems and deep learning ● NLP ○ Medical dialogue summarization ○ Transfer Learning for Similar Questions ○ Medical entity recognition: An active learning approach ● Multimodal healthcare AI ○ Few-shot dermatology image classification
  • 36. General principles 1. Extensible a. Data feedback loops b. Incrementally/iteratively learn from “physician-in-the-loop” 2. Domain knowledge + Data 3. In the wild a. Uncertainty in prediction b. Fall-back to “physician-in-the-loop”
  • 38. SOTA Medical reasoning and diagnosis
  • 39. Our approach to COVID-19
  • 40. ML + Expert systems for Dx models female middle aged fever cough Influenza 16.9 bacterial pneumonia 16.9 acute sinusitis 10.9 asthma 10.9 common cold 10.9 influenza 0.753 bacterial pneumonia 0.205 asthma 0.017 acute sinusitis 0.008 pulmonary tuberculosis 0.007 Inputs DDx with expert system DDx with ML model Expert system Clinical case simulator Clinical cases DDx ML model Common cold UTI Acute bronchitis Female Middle-aged Chronic cough Nasal congestion Other data (e.g. EHR)
  • 41. COVID-aware modeling Expert system Clinical case simulator Clinical cases with DDx ML model Common cold UTI Acute bronchitis Female Middle-aged Chronic cough Nasal congestion COVID-19 assessment data COVID-19 COVID-19 female middle-age cough headache nose discharge cigarette smoking hospital personnel
  • 42. Examples Inputs DDx before COVID DDx after COVID female middle aged fever cough healthcare worker influenza bacterial pneumonia asthma COVID-19 influenza bacterial pneumonia female middle aged fever cough nasal congestion influenza adenovirus infection bacterial pneumonia influenza COVID-19 adenovirus infection
  • 43. Question similarity for COVID FAQs ● Transfer learning ● Double-finetuned BERT model ● Handle data sparsity ● Medical domain knowledge through an intermediate QA binary task Does A answer Q? Q A Pretrained BERT BERT Q1 similar to Q2? Q1 Q2 BERT Proceedings of the 2020 ACM SIGKDD
  • 45. Medical summarization Best paper at ACL Workshop on medical conversations 2021 Machine Learning for Healthcare, 2021 EMNLP Findings 2020
  • 46. Our Approach: GPT-3-ENS Hasn't used any thing to help GPT-3 Priming Inference GPT-3 GPT-3 Hasn’t used any thing to help other than hydrocortisone Used nothing else to help other than Benadryl and hydrocortisone. 10 Trials 21 Unique Labeled Examples per Priming Context Confidentiality note: In accordance with our privacy policy, the illustrative examples included in this document do NOT correspond to real patients. They are either synthetic or fully anonymized. Chat Snippet: DR: Thanks for answering my questions. Apart from Benadryl and Hydrocortisone, have you used anything to help? PT: No that’s everything
  • 47. GPT-3 vs GPT-3-ENS ● 12.5% of summaries produced by GPT-3-ENS are better than those of GPT-3
  • 48. GPT-3-ENS as data synthesizer Hasn't used any thing to help Priming Inference GPT-3 Hasn’t used any thing to help other than hydrocortisone Used nothing else to help other than Benadryl and hydrocortisone. 10 Trials 21 Labeled Examples per Priming Context GPT-3 GPT-3 GPT-3-ENS Labeled Dataset + Doctor Labeled/Corrected Dataset In-House Summarization Model ● Privacy concerns with invoking GPT-3 at inference time ● Doctor in the loop Confidentiality note: In accordance with our privacy policy, the illustrative examples included in this document do NOT correspond to real patients. They are either synthetic or fully anonymized. Chat Snippet: DR: Thanks for ... PT: No that’s everything
  • 49. Results ● We are able to train a summarization model using GPT-3-ENS labeled data (which needed 210 doctor-labeled examples) comparable in performance to a model trained using 6400 doctor-labeled examples ● We find that a model that is trained on a mix of human and GPT-3-ENS labeled data does better than a model trained on either independently
  • 50. Conclusions ● Healthcare needs to scale quickly, and this has become obvious in a global pandemic like the one we are facing ● The only way to scale healthcare while improving quality and accessibility is through technology and SOTA AI ● But, AI cannot be simply “dropped” in the middle of old workflows and processes ○ It needs to be integrated in end-to-end medical care benefitting both patients and providers https://curai.com/work
  • 51. Outline 1. The recommender problem & the Netflix Prize 2. Recommendations beyond rating prediction 3. AI + Healthcare: A bit about Curai 4. Lessons Learned
  • 53. 1.Data or and Models?
  • 54. More data or better models? Really?
  • 55. More data or better models? Sometimes, it’s not about more data
  • 56. More data or better models? Norvig: “Google does not have better Algorithms only more Data” Many features/ low-bias models
  • 57. More data or better models? Sometimes you might not need all your “Big Data” 0 2 4 6 8 10 12 14 16 18 20 Number of Training Examples (in Millions) Testing Accuracy
  • 58. What about Deep Learning? Year Breakthrough in AI Datasets (First Available) Algorithms (First Proposal) 1994 Human-level spontaneous speech recognition Spoken Wall Street Journal articles and other texts (1991) Hidden Markov Model (1984) 1997 IBM Deep Blue defeated Garry Kasparov 700,000 Grandmaster chess games, aka “The Extended Book” (1991) Negascout planning algorithm (1983) 2005 Google’s Arabic- and Chinese-to-English translation 1,8 trillion tokens from Google Web and News pages (collected in 2005) Statistical machine translation algorithm (1988) 2011 IBM watson become the world Jeopardy! Champion 8,6 million documents from Wikipedia, Wiktionary, Wikiquote, and Project Gutenberg (updated in 2005) Mixture-of-Experts algorithm (1991) 2014 Google’s GoogLeNet object classification at near-human performance ImageNet corpus of 1,5 million labeled images and 1,000 object catagories (2010) Convolution neural network algorithm (1989) 2015 Google’s Deepmind achieved human parity in playing 29 Atari games by learning general control from video Arcade Learning Environment dataset of over 50 Atari games (2013) Q-learning algorithm (1992) Average No. Of Years to Breakthrough 3 years 18 years The average elapsed time between key algorithm proposals and corresponding advances was about 18 years, whereas the average elapsed time between key dataset availabilities and corresponding advances was less than 3 years, or about 6 times faster.
  • 59. Models and Recipes Pretrained Available models trained using OpenNMT → English → German → German → English → English Summarization → Multi-way – FR,ES,PT,IT,RO < > FR,ES,PT,IT,RO More models coming soon: → Ubuntu Dialog Dataset → Syntactic Parsing → Image-to-Text What about Deep Learning?
  • 60. More data and better data
  • 61. 2.Implicit user signals Are Always sometimes better than Explicit ones
  • 62. Implicit vs. Explicit ● Many have acknowledged that implicit feedback is more useful ● Is implicit feedback really always more useful? ● If so, why?
  • 63. ● Implicit data is (usually): ○ More dense, and available for all users ○ Better representative of user behavior vs. user reflection ○ More related to final objective function ○ Better correlated with AB test results ● E.g. Rating vs watching Implicit vs. Explicit
  • 64. ● However ○ It is not always the case that direct implicit feedback correlates well with long-term retention ○ E.g. clickbait ● Solution: ○ Combine different forms of implicit + explicit to better represent long-term goal Implicit vs. Explicit
  • 65. 3. MODEL ACCURACY IS NOT ALL THAT MATTERS
  • 67. 4. Model training depends not only on input data
  • 68. Training a model ● Model will learn according to: ○ Training data (e.g. implicit and explicit) ○ Target function (e.g. probability of user reading an answer) ○ Metric (e.g. precision vs. recall) ● Example 1 (made up): ○ Optimize probability of a user going to the cinema to watch a movie and rate it “highly” by using purchase history and previous ratings. Use NDCG of the ranking as final metric using only movies rated 4 or higher as positives.
  • 69. Example 2 - Quora’s feed ● Training data = implicit + explicit ● Target function: Value of showing a story to a user ~ weighted sum of actions: v = ∑a va 1{ya = 1} ○ predict probabilities for each action, then compute expected value: v_pred = E[ V | x ] = ∑a va p(a | x) ● Metric: any ranking metric
  • 70. 5. THERE IS SOMETHING EVEN MORE IMPORTANT THAN MODELS AND DATA: EXPERIMENTAL PROCESS
  • 72. Executing A/B tests ● Measure differences in metrics across statistically identical populations that each experience a different algorithm. ● Decisions on the product always data-driven ● Overall Evaluation Criteria (OEC) = member retention ○ Use long-term metrics whenever possible ○ Short-term metrics can be informative and allow faster decisions ■ But, not always aligned with OEC
  • 73. Offline testing ● Measure model performance, using (IR) metrics ● Offline performance = indication to make decisions on follow-up A/B tests ● A critical (and mostly unsolved) issue is how offline metrics correlate with A/B test results.
  • 75. 01. 02. 03. 04. 05. Choose the right metric Be thoughtful about your data Understand dependencies between data, models & systems Optimize only what matters, beware of biases Be thoughtful about : data infrastructure/tools, how to organize your teams, and above all, your product requirements
  • 77. 4 hour lecture on recommendations Carnegie Mellon (2014) 1 hour lecture on practical Deep Learning UC Berkeley (2020) 10 minutes on AI for COVID Stanford (2020) 1 hour podcast on AI for Healthcare Gradient Dissent (2021)