SlideShare a Scribd company logo
1 of 35
Download to read offline
Understanding How People Use
Natural Language to Ask for
Recommendations
Jie Kang*, Kyle Condiff*, Shuo Chang**, Joseph A. Konstan,
Loren Terveen, Max Harper (presenter)
1
GroupLens Center for Social and Human-Centered Computing
University of Minnesota
* Now with Facebook
** Now with Quora
Overview of this Talk:
Natural Language Recommenders
» motivation: why natural language
recommenders are cool
» experiment: lab experiment + qualitative
analysis
» discussion: dataset, design implications,
opportunities
2
Overview of this Talk:
Natural Language Recommenders
» motivation: why natural language
recommenders are cool
» experiment: lab experiment + qualitative
analysis
» discussion: dataset, design implications,
opportunities
3
4https://www.flickr.com/photos/group3planners/5314810474/
librarian recommender flow
» me: seed
recommendations
with items
» librarian: offer
suggestions
» me: critique (“too
scary”) or accept
suggestions
5
librarian as recommender interface
» I can seed the conversation in a natural
way (ask for what you want!)
» I can detect that the conversation is going
astray, and correct (too scary! too old!)
» I can be vague in my query, or very
specific, depending on my mood
6
vs. canonical recommender UI
» endless lists, best
stuff tries to be at the
top
» sometimes based on
recent activity
(“context”)
» downsides?
7
a technological gap
natural language tech recommendation tech
8
bridging the gap: voice control
» voice interfaces, e.g.:
• Amazon Fire as video
player
• Google Home as
music player
» voice recognition
getting better
» goal: better
integration with
recommender
technologies
9
bridging the gap: chatbots
» chat interfaces, e.g.:
• And Chill on Facebook
Messenger
• LunchBot on Slack
» frameworks to build
these dialogues (e.g.,
wit.ai) are very
accessible
» goal: richer, more
flexible dialogue
10
GOAL: BUILD NATURAL
LANGUAGE INTERFACES TO
RECOMMENDERS THAT
REASONABLY APPROXIMATE
THE LIBRARIAN EXPERIENCE
11
Overview of this Talk:
Natural Language Recommenders
» motivation: why natural language
recommenders are cool
» experiment: lab experiment + qualitative
analysis
» discussion: dataset, design implications,
opportunities
12
DESIGN PROBLEM: WHAT
WILL USERS ASK FOR?
13
before we can ask “how do we respond to
natural language recommendation
requests?” we must ask the following
research question:
» how do users ask for recommendations
and express their preferences using
natural language?
14
experiment overview
» collect dataset of queries
• recruit MovieLens users by email
• assign subjects to speaking and typing
conditions
• collect queries and survey responses
» qualitatively code queries
first query (N=347)
speaking typing
16
(speech to text by wit.ai)
follow-up query (N=151)
» show 10 recs
» “I can improve
these results. Tell
me more about
what you want.”
» same
speaking/typing
UI as first query
17
extracting meaning from queries
» Inspired by Rose and Levinson (WWW 2004): goals of
users in search (navigational, informational, resource)
» inductive, open coding
• four researchers read through the dataset, iteratively
assign new codes and refine old codes until stable
• final codes were consensus of two researchers who
discussed and resolved disagreements
» evaluation of coding consistency
• two researchers coded 187 random queries to measure
consistency
• Cohen’s kappa 0.87
18
RESULTS
19
first queries:
three top-level features
» objective
» subjective
» navigation
20
objective
genre “superhero movies”
deep features “movies with open endings or plot twists”
people “Brad Pitt”
release date “can you find me a funny romantic movie
made in the 2000s?”
region “British murder mystery”
language “show me a list of German movies”
21
» known attributes
» filtering
subjective
emotion “sad movie”
quality “interesting characters, clever plot”
movie-based “what would you recommend to a fan of
Big Lebowski?”
22
» quality judgments
» ordering
navigation
» go directly to an item
» “blade runner”
23
24
233
77 81
25
follow-up queries: three types
» refine
» reformulate
» start over
26
follow-up query type: refine
Refine with
further
constraints
1: a mystery drama with a suspenseful
ending
2: something from the last few years
Refine with
clarification
1: Horror
2: More true horror instead of drama/
thriller
27
» specify additional criteria for the
recommender to consider
follow-up query type: reformulate
Reformulate with
further
constraints
1: I’m looking for a romantic comedy
2: I’d like a romantic comedy that was
created after the year
2000
Reformulate with
clarification
1: a romantic comedy with a happy
ending
2: romantic comedy with tensions
between the couple but ends
well
28
» completely restate the query
follow-up query type: start over
29
1. mad max fury road
2. Finding dory
text vs. speech
» speaking: longer queries
» speaking: more likely “conversational”
» speaking: more queries with objective
deep features and subjective queries
30
Overview of this Talk:
Natural Language Recommenders
» motivation: why natural language
recommenders are cool
» experiment: lab experiment + qualitative
analysis
» discussion: dataset, design implications,
opportunities
31
design implications
» objective deep features and subjective
features: important (and difficult?) to
support
» detect and differentially support objective
(filter) vs. subjective (sort)
» detect and support the “intent” of follow-up
(critiquing) queries
32
opportunities
» more than movies! how do people interact
differently in different domains?
» different UIs: voice-only, voice + screen
(Amazon Echo), typing (Facebook
Messenger)
» conversational recommenders
» decision-making with intelligent agents
33
dataset
34
» queries, survey responses
» link in paper (or search for “understanding how
people use natural language to ask for
recommendations”)
Understanding How People Use
Natural Language to Ask for
Recommendations
Max Harper
Research Scientist
GroupLens Research, University of Minnesota
max@umn.edu
@maxharp3r
35
This material is based on work supported by the National Science Foundation under grants
IIS-0964695, IIS-1017697, IIS-1111201, IIS- 1210863, and IIS-1218826, and by a grant from
Google.

More Related Content

Similar to Recsys 2017 -- Understanding How People Use Natural Language to Ask for Recommendations

Using Surveys to Improve Your Library: Part 2 (Sept. 2018)
Using Surveys to Improve Your Library: Part 2 (Sept. 2018)Using Surveys to Improve Your Library: Part 2 (Sept. 2018)
Using Surveys to Improve Your Library: Part 2 (Sept. 2018)ALATechSource
 
Technology intergrationplan kellieouzts_application
Technology intergrationplan kellieouzts_applicationTechnology intergrationplan kellieouzts_application
Technology intergrationplan kellieouzts_applicationBarrow County Schools
 
xCulture – Cross-cultural UX Elements: Research Method and Design Guidelines
xCulture – Cross-cultural UX Elements: Research Method and Design GuidelinesxCulture – Cross-cultural UX Elements: Research Method and Design Guidelines
xCulture – Cross-cultural UX Elements: Research Method and Design GuidelinesJan Brejcha
 
1. An outline is a tool for organizing online web searches. (Poi.docx
1. An outline is a tool for organizing online web searches. (Poi.docx1. An outline is a tool for organizing online web searches. (Poi.docx
1. An outline is a tool for organizing online web searches. (Poi.docxcorbing9ttj
 
How machines learn to talk. Machine Learning for Conversational AI
How machines learn to talk. Machine Learning for Conversational AIHow machines learn to talk. Machine Learning for Conversational AI
How machines learn to talk. Machine Learning for Conversational AIVerena Rieser
 
Quantifying reflection
Quantifying reflectionQuantifying reflection
Quantifying reflectionfridolin.wild
 
Reference interviews
Reference interviewsReference interviews
Reference interviewsYongtao Lin
 
Researching people: using questionnaires and interviews
Researching people: using questionnaires and interviewsResearching people: using questionnaires and interviews
Researching people: using questionnaires and interviewsJenna Condie
 
Week 11 english 145
Week 11 english 145 Week 11 english 145
Week 11 english 145 lisyaseloni
 
Using Surveys to Improve Your Library - Part 2
Using Surveys to Improve Your Library - Part 2Using Surveys to Improve Your Library - Part 2
Using Surveys to Improve Your Library - Part 2ALATechSource
 
Multimedia Academic Literacy
Multimedia Academic LiteracyMultimedia Academic Literacy
Multimedia Academic LiteracySpelman College
 
Tips for collecting_high_quality_qualitative_data
Tips for collecting_high_quality_qualitative_dataTips for collecting_high_quality_qualitative_data
Tips for collecting_high_quality_qualitative_dataSC CTSI at USC and CHLA
 
Giving great talks in Bioinformatics - from Professional Communication class ...
Giving great talks in Bioinformatics - from Professional Communication class ...Giving great talks in Bioinformatics - from Professional Communication class ...
Giving great talks in Bioinformatics - from Professional Communication class ...Ann Loraine
 
Developing a Mixed Qualitative and Quantitative Research Design to Inform Lib...
Developing a Mixed Qualitative and Quantitative Research Design to Inform Lib...Developing a Mixed Qualitative and Quantitative Research Design to Inform Lib...
Developing a Mixed Qualitative and Quantitative Research Design to Inform Lib...Marie Kennedy
 
Speech act-andrew-d-cohen-1206263596754620-5
Speech act-andrew-d-cohen-1206263596754620-5Speech act-andrew-d-cohen-1206263596754620-5
Speech act-andrew-d-cohen-1206263596754620-5xiongying
 
FSU SLIS Wk2 Intro to Info Services: Reference Interview
FSU SLIS Wk2 Intro to Info Services: Reference InterviewFSU SLIS Wk2 Intro to Info Services: Reference Interview
FSU SLIS Wk2 Intro to Info Services: Reference InterviewLorri Mon
 
Chapter 13 methods of discovery
Chapter 13   methods of discoveryChapter 13   methods of discovery
Chapter 13 methods of discoveryxiaoshandanni
 

Similar to Recsys 2017 -- Understanding How People Use Natural Language to Ask for Recommendations (20)

Using Surveys to Improve Your Library: Part 2 (Sept. 2018)
Using Surveys to Improve Your Library: Part 2 (Sept. 2018)Using Surveys to Improve Your Library: Part 2 (Sept. 2018)
Using Surveys to Improve Your Library: Part 2 (Sept. 2018)
 
Technology intergrationplan kellieouzts_application
Technology intergrationplan kellieouzts_applicationTechnology intergrationplan kellieouzts_application
Technology intergrationplan kellieouzts_application
 
xCulture – Cross-cultural UX Elements: Research Method and Design Guidelines
xCulture – Cross-cultural UX Elements: Research Method and Design GuidelinesxCulture – Cross-cultural UX Elements: Research Method and Design Guidelines
xCulture – Cross-cultural UX Elements: Research Method and Design Guidelines
 
Info521 week2
Info521 week2Info521 week2
Info521 week2
 
1. An outline is a tool for organizing online web searches. (Poi.docx
1. An outline is a tool for organizing online web searches. (Poi.docx1. An outline is a tool for organizing online web searches. (Poi.docx
1. An outline is a tool for organizing online web searches. (Poi.docx
 
2106 JWLLP
2106 JWLLP2106 JWLLP
2106 JWLLP
 
How machines learn to talk. Machine Learning for Conversational AI
How machines learn to talk. Machine Learning for Conversational AIHow machines learn to talk. Machine Learning for Conversational AI
How machines learn to talk. Machine Learning for Conversational AI
 
Quantifying reflection
Quantifying reflectionQuantifying reflection
Quantifying reflection
 
Reference interviews
Reference interviewsReference interviews
Reference interviews
 
Researching people: using questionnaires and interviews
Researching people: using questionnaires and interviewsResearching people: using questionnaires and interviews
Researching people: using questionnaires and interviews
 
Week 11 english 145
Week 11 english 145 Week 11 english 145
Week 11 english 145
 
Using Surveys to Improve Your Library - Part 2
Using Surveys to Improve Your Library - Part 2Using Surveys to Improve Your Library - Part 2
Using Surveys to Improve Your Library - Part 2
 
Multimedia Academic Literacy
Multimedia Academic LiteracyMultimedia Academic Literacy
Multimedia Academic Literacy
 
Dove, "A Model of the User's Psychological State as a Framework for Understan...
Dove, "A Model of the User's Psychological State as a Framework for Understan...Dove, "A Model of the User's Psychological State as a Framework for Understan...
Dove, "A Model of the User's Psychological State as a Framework for Understan...
 
Tips for collecting_high_quality_qualitative_data
Tips for collecting_high_quality_qualitative_dataTips for collecting_high_quality_qualitative_data
Tips for collecting_high_quality_qualitative_data
 
Giving great talks in Bioinformatics - from Professional Communication class ...
Giving great talks in Bioinformatics - from Professional Communication class ...Giving great talks in Bioinformatics - from Professional Communication class ...
Giving great talks in Bioinformatics - from Professional Communication class ...
 
Developing a Mixed Qualitative and Quantitative Research Design to Inform Lib...
Developing a Mixed Qualitative and Quantitative Research Design to Inform Lib...Developing a Mixed Qualitative and Quantitative Research Design to Inform Lib...
Developing a Mixed Qualitative and Quantitative Research Design to Inform Lib...
 
Speech act-andrew-d-cohen-1206263596754620-5
Speech act-andrew-d-cohen-1206263596754620-5Speech act-andrew-d-cohen-1206263596754620-5
Speech act-andrew-d-cohen-1206263596754620-5
 
FSU SLIS Wk2 Intro to Info Services: Reference Interview
FSU SLIS Wk2 Intro to Info Services: Reference InterviewFSU SLIS Wk2 Intro to Info Services: Reference Interview
FSU SLIS Wk2 Intro to Info Services: Reference Interview
 
Chapter 13 methods of discovery
Chapter 13   methods of discoveryChapter 13   methods of discovery
Chapter 13 methods of discovery
 

Recently uploaded

Role of Gibberellins, mode of action and external applications.pptx
Role of Gibberellins, mode of action and external applications.pptxRole of Gibberellins, mode of action and external applications.pptx
Role of Gibberellins, mode of action and external applications.pptxjana861314
 
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀Tokyo Chemicals Industry (TCI)
 
Potato spindle tuber disease VIRIOID DISEASE
Potato spindle tuber disease  VIRIOID DISEASEPotato spindle tuber disease  VIRIOID DISEASE
Potato spindle tuber disease VIRIOID DISEASEKARTHIK REDDY C A
 
Monggo Seeds Experiment Day 1 to Day 7.pdf
Monggo Seeds Experiment Day 1 to Day 7.pdfMonggo Seeds Experiment Day 1 to Day 7.pdf
Monggo Seeds Experiment Day 1 to Day 7.pdfRoseAnnAquino8
 
FBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptxFBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptxPayal Shrivastava
 
Think Science: What Are Eclipses, by Craig Bobchin
Think Science: What Are Eclipses, by Craig BobchinThink Science: What Are Eclipses, by Craig Bobchin
Think Science: What Are Eclipses, by Craig BobchinNathan Cone
 
chapter 28 Reproductive System Power Point
chapter 28 Reproductive System Power Pointchapter 28 Reproductive System Power Point
chapter 28 Reproductive System Power Pointchelseakland
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaDr.Mahmoud Abbas
 
Efficient Fourier Pricing of Multi-Asset Options: Quasi-Monte Carlo & Domain ...
Efficient Fourier Pricing of Multi-Asset Options: Quasi-Monte Carlo & Domain ...Efficient Fourier Pricing of Multi-Asset Options: Quasi-Monte Carlo & Domain ...
Efficient Fourier Pricing of Multi-Asset Options: Quasi-Monte Carlo & Domain ...Chiheb Ben Hammouda
 
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By KarishmaSTELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By KarishmaAMiracle3
 
Timeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological CorrelationsTimeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological CorrelationsDanielBaumann11
 
Interpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWSTInterpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWSTAlexander F. Mayer
 
Mycoplasma Mycoplasma pneumoniae K R.pptx
Mycoplasma Mycoplasma pneumoniae  K R.pptxMycoplasma Mycoplasma pneumoniae  K R.pptx
Mycoplasma Mycoplasma pneumoniae K R.pptxKARTHIK REDDY C A
 
DEVIATION OF REAL GAS FROM IDEAL BEHAVIOUR.pptx
DEVIATION OF REAL GAS FROM IDEAL BEHAVIOUR.pptxDEVIATION OF REAL GAS FROM IDEAL BEHAVIOUR.pptx
DEVIATION OF REAL GAS FROM IDEAL BEHAVIOUR.pptxAiswarya47
 
Rabies ,a deadly viral disease transmitted by the most beautiful beings. by ...
Rabies ,a deadly viral disease transmitted by the most beautiful beings.  by ...Rabies ,a deadly viral disease transmitted by the most beautiful beings.  by ...
Rabies ,a deadly viral disease transmitted by the most beautiful beings. by ...uzmashireenmbe01
 
Physical pharmacy unit 1 solubility of drugs.pptx
Physical pharmacy unit 1 solubility of drugs.pptxPhysical pharmacy unit 1 solubility of drugs.pptx
Physical pharmacy unit 1 solubility of drugs.pptxchetanvgh
 
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptxPharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptxeAuditor Audits & Inspections
 
FUNDAMENTAL OF ENTOMOLOGY notes for b sc agriculture
FUNDAMENTAL OF ENTOMOLOGY notes for b sc agricultureFUNDAMENTAL OF ENTOMOLOGY notes for b sc agriculture
FUNDAMENTAL OF ENTOMOLOGY notes for b sc agricultureDrAnita Sharma
 
Astrology's Guidance in Decision Making.
Astrology's Guidance in Decision Making.Astrology's Guidance in Decision Making.
Astrology's Guidance in Decision Making.astral oracle
 
Think Science: What Are Eclipses (101), by Craig Bobchin
Think Science: What Are Eclipses (101), by Craig BobchinThink Science: What Are Eclipses (101), by Craig Bobchin
Think Science: What Are Eclipses (101), by Craig BobchinNathan Cone
 

Recently uploaded (20)

Role of Gibberellins, mode of action and external applications.pptx
Role of Gibberellins, mode of action and external applications.pptxRole of Gibberellins, mode of action and external applications.pptx
Role of Gibberellins, mode of action and external applications.pptx
 
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
 
Potato spindle tuber disease VIRIOID DISEASE
Potato spindle tuber disease  VIRIOID DISEASEPotato spindle tuber disease  VIRIOID DISEASE
Potato spindle tuber disease VIRIOID DISEASE
 
Monggo Seeds Experiment Day 1 to Day 7.pdf
Monggo Seeds Experiment Day 1 to Day 7.pdfMonggo Seeds Experiment Day 1 to Day 7.pdf
Monggo Seeds Experiment Day 1 to Day 7.pdf
 
FBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptxFBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptx
 
Think Science: What Are Eclipses, by Craig Bobchin
Think Science: What Are Eclipses, by Craig BobchinThink Science: What Are Eclipses, by Craig Bobchin
Think Science: What Are Eclipses, by Craig Bobchin
 
chapter 28 Reproductive System Power Point
chapter 28 Reproductive System Power Pointchapter 28 Reproductive System Power Point
chapter 28 Reproductive System Power Point
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
 
Efficient Fourier Pricing of Multi-Asset Options: Quasi-Monte Carlo & Domain ...
Efficient Fourier Pricing of Multi-Asset Options: Quasi-Monte Carlo & Domain ...Efficient Fourier Pricing of Multi-Asset Options: Quasi-Monte Carlo & Domain ...
Efficient Fourier Pricing of Multi-Asset Options: Quasi-Monte Carlo & Domain ...
 
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By KarishmaSTELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
 
Timeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological CorrelationsTimeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
 
Interpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWSTInterpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWST
 
Mycoplasma Mycoplasma pneumoniae K R.pptx
Mycoplasma Mycoplasma pneumoniae  K R.pptxMycoplasma Mycoplasma pneumoniae  K R.pptx
Mycoplasma Mycoplasma pneumoniae K R.pptx
 
DEVIATION OF REAL GAS FROM IDEAL BEHAVIOUR.pptx
DEVIATION OF REAL GAS FROM IDEAL BEHAVIOUR.pptxDEVIATION OF REAL GAS FROM IDEAL BEHAVIOUR.pptx
DEVIATION OF REAL GAS FROM IDEAL BEHAVIOUR.pptx
 
Rabies ,a deadly viral disease transmitted by the most beautiful beings. by ...
Rabies ,a deadly viral disease transmitted by the most beautiful beings.  by ...Rabies ,a deadly viral disease transmitted by the most beautiful beings.  by ...
Rabies ,a deadly viral disease transmitted by the most beautiful beings. by ...
 
Physical pharmacy unit 1 solubility of drugs.pptx
Physical pharmacy unit 1 solubility of drugs.pptxPhysical pharmacy unit 1 solubility of drugs.pptx
Physical pharmacy unit 1 solubility of drugs.pptx
 
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptxPharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
Pharmaceutical warehouse audit using eAuditor Audits & Inspections .pptx
 
FUNDAMENTAL OF ENTOMOLOGY notes for b sc agriculture
FUNDAMENTAL OF ENTOMOLOGY notes for b sc agricultureFUNDAMENTAL OF ENTOMOLOGY notes for b sc agriculture
FUNDAMENTAL OF ENTOMOLOGY notes for b sc agriculture
 
Astrology's Guidance in Decision Making.
Astrology's Guidance in Decision Making.Astrology's Guidance in Decision Making.
Astrology's Guidance in Decision Making.
 
Think Science: What Are Eclipses (101), by Craig Bobchin
Think Science: What Are Eclipses (101), by Craig BobchinThink Science: What Are Eclipses (101), by Craig Bobchin
Think Science: What Are Eclipses (101), by Craig Bobchin
 

Recsys 2017 -- Understanding How People Use Natural Language to Ask for Recommendations

  • 1. Understanding How People Use Natural Language to Ask for Recommendations Jie Kang*, Kyle Condiff*, Shuo Chang**, Joseph A. Konstan, Loren Terveen, Max Harper (presenter) 1 GroupLens Center for Social and Human-Centered Computing University of Minnesota * Now with Facebook ** Now with Quora
  • 2. Overview of this Talk: Natural Language Recommenders » motivation: why natural language recommenders are cool » experiment: lab experiment + qualitative analysis » discussion: dataset, design implications, opportunities 2
  • 3. Overview of this Talk: Natural Language Recommenders » motivation: why natural language recommenders are cool » experiment: lab experiment + qualitative analysis » discussion: dataset, design implications, opportunities 3
  • 5. librarian recommender flow » me: seed recommendations with items » librarian: offer suggestions » me: critique (“too scary”) or accept suggestions 5
  • 6. librarian as recommender interface » I can seed the conversation in a natural way (ask for what you want!) » I can detect that the conversation is going astray, and correct (too scary! too old!) » I can be vague in my query, or very specific, depending on my mood 6
  • 7. vs. canonical recommender UI » endless lists, best stuff tries to be at the top » sometimes based on recent activity (“context”) » downsides? 7
  • 8. a technological gap natural language tech recommendation tech 8
  • 9. bridging the gap: voice control » voice interfaces, e.g.: • Amazon Fire as video player • Google Home as music player » voice recognition getting better » goal: better integration with recommender technologies 9
  • 10. bridging the gap: chatbots » chat interfaces, e.g.: • And Chill on Facebook Messenger • LunchBot on Slack » frameworks to build these dialogues (e.g., wit.ai) are very accessible » goal: richer, more flexible dialogue 10
  • 11. GOAL: BUILD NATURAL LANGUAGE INTERFACES TO RECOMMENDERS THAT REASONABLY APPROXIMATE THE LIBRARIAN EXPERIENCE 11
  • 12. Overview of this Talk: Natural Language Recommenders » motivation: why natural language recommenders are cool » experiment: lab experiment + qualitative analysis » discussion: dataset, design implications, opportunities 12
  • 13. DESIGN PROBLEM: WHAT WILL USERS ASK FOR? 13
  • 14. before we can ask “how do we respond to natural language recommendation requests?” we must ask the following research question: » how do users ask for recommendations and express their preferences using natural language? 14
  • 15. experiment overview » collect dataset of queries • recruit MovieLens users by email • assign subjects to speaking and typing conditions • collect queries and survey responses » qualitatively code queries
  • 16. first query (N=347) speaking typing 16 (speech to text by wit.ai)
  • 17. follow-up query (N=151) » show 10 recs » “I can improve these results. Tell me more about what you want.” » same speaking/typing UI as first query 17
  • 18. extracting meaning from queries » Inspired by Rose and Levinson (WWW 2004): goals of users in search (navigational, informational, resource) » inductive, open coding • four researchers read through the dataset, iteratively assign new codes and refine old codes until stable • final codes were consensus of two researchers who discussed and resolved disagreements » evaluation of coding consistency • two researchers coded 187 random queries to measure consistency • Cohen’s kappa 0.87 18
  • 20. first queries: three top-level features » objective » subjective » navigation 20
  • 21. objective genre “superhero movies” deep features “movies with open endings or plot twists” people “Brad Pitt” release date “can you find me a funny romantic movie made in the 2000s?” region “British murder mystery” language “show me a list of German movies” 21 » known attributes » filtering
  • 22. subjective emotion “sad movie” quality “interesting characters, clever plot” movie-based “what would you recommend to a fan of Big Lebowski?” 22 » quality judgments » ordering
  • 23. navigation » go directly to an item » “blade runner” 23
  • 25. 25
  • 26. follow-up queries: three types » refine » reformulate » start over 26
  • 27. follow-up query type: refine Refine with further constraints 1: a mystery drama with a suspenseful ending 2: something from the last few years Refine with clarification 1: Horror 2: More true horror instead of drama/ thriller 27 » specify additional criteria for the recommender to consider
  • 28. follow-up query type: reformulate Reformulate with further constraints 1: I’m looking for a romantic comedy 2: I’d like a romantic comedy that was created after the year 2000 Reformulate with clarification 1: a romantic comedy with a happy ending 2: romantic comedy with tensions between the couple but ends well 28 » completely restate the query
  • 29. follow-up query type: start over 29 1. mad max fury road 2. Finding dory
  • 30. text vs. speech » speaking: longer queries » speaking: more likely “conversational” » speaking: more queries with objective deep features and subjective queries 30
  • 31. Overview of this Talk: Natural Language Recommenders » motivation: why natural language recommenders are cool » experiment: lab experiment + qualitative analysis » discussion: dataset, design implications, opportunities 31
  • 32. design implications » objective deep features and subjective features: important (and difficult?) to support » detect and differentially support objective (filter) vs. subjective (sort) » detect and support the “intent” of follow-up (critiquing) queries 32
  • 33. opportunities » more than movies! how do people interact differently in different domains? » different UIs: voice-only, voice + screen (Amazon Echo), typing (Facebook Messenger) » conversational recommenders » decision-making with intelligent agents 33
  • 34. dataset 34 » queries, survey responses » link in paper (or search for “understanding how people use natural language to ask for recommendations”)
  • 35. Understanding How People Use Natural Language to Ask for Recommendations Max Harper Research Scientist GroupLens Research, University of Minnesota max@umn.edu @maxharp3r 35 This material is based on work supported by the National Science Foundation under grants IIS-0964695, IIS-1017697, IIS-1111201, IIS- 1210863, and IIS-1218826, and by a grant from Google.