Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Parts 1 & 2: WWW 2018 Tutorial: Understanding User Needs & Tasks

75 views

Published on

WWW 2018 tutorial on Understanding User Needs & Tasks
Details: https://task-ir.github.io/Task-based-Search/

Published in: Software
  • Be the first to comment

Parts 1 & 2: WWW 2018 Tutorial: Understanding User Needs & Tasks

  1. 1. Inferring User Tasks and Needs Rishabh Mehrotra1, Emine Yilmaz2, Ahmed Hassan Awadallah3 1Spotify, London 2University College London 3Microsoft Research
  2. 2. Phase I: Understanding User Tasks & Needs 1. What is a task & why are they important? 2. Characterizing Tasks across devices & interfaces: desktop search, digital assistants & voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks c. Evaluating task extraction algorithms 4. Recommendation Systems a. User intents & goals b. Defining user intents c. Predicting user intents
  3. 3. Section 1: Introduction
  4. 4. Introduction Search is Everywhere! Understanding users’ needs is HARD!
  5. 5. Why “Tasks” are Important? • Long sessions are common – 40% of sessions contain multiple queries – 30% of sessions take 5+ minutes – 7% of these take 30+ minutes 71% 12% 10% 7% Percentage of Sesions of Different Time Lengths < 5 mins 5 – 15 mins 15 – 30 mins > 30 mins [Dumais, NSF Search Tasks Workshop 2013]
  6. 6. • Long sessions account for most of the search time – 50% of time spent in sessions of 30+ mins 4% 15% 29% 52% Time in Sessions of Different Time Lengths < 5 mins 5 – 15 mins 15 – 30 mins > 30 mins Why “Tasks” are Important? [Dumais, NSF Search Tasks Workshop 2013]
  7. 7. • Long sessions are more likely to continue – Probability of issuing another query, given session of length N-1 Why “Tasks” are Important? [Bailey et al., NII Shannon 2012]
  8. 8. People use search to accomplish variety of task [Bailey et al., NII Shannon 2012]
  9. 9. Task Minutes per task (Avg.) Queries per task (Avg.) Compare products/ services 25 7 Plan travel 12 5 Find real estate 15 5 Buy a product 9 3 Many tasks require significant effort to achieve [Bailey et al., NII Shannon 2012]
  10. 10. What is a “Task”? Online credit check Mortgage in principle Facebook House buying guide Quit smoking benefits Solicitors near me … Houses for sale Loans for house … 10:00am 10:03am 10:07am 12:30pm 17:00pm 17:02pm 17:06pm 18:15pm Session 1 Session 2 Session 3 Session 4 A task is an atomic information need resulting in one or more queries [Jones and Klinkner, CIKM 2008]
  11. 11. Why identify “Tasks”?
  12. 12. Outline of the Tutorial • Section 1: Introduction • Section 2: Characterizing Tasks • Section 3: Tasks Extraction Algorithms • Section 4: Task based Evaluation • Section 5: Applications
  13. 13. Section 2: Characterizing Tasks • Understanding Intents & Tasks – Query intents in IR – Search sessions – Sessions à Tasks • Characterizing Tasks across devices – Desktop based search • Taxonomy of browsing & querying behavior – Assistants • Digital Assistants – Use cases – User engagement • Voice-only Assistants – Use cases – User engagement
  14. 14. Query Intents in IR • Queries often – carry some degree of ambiguity – are underspecified, or – have various aspects/interpretations • Identifying & understanding the intents helps distinguish ambiguous or multi-faceted queries.
  15. 15. • Query classification aims to classify queries into pre- defined intents Pre-defined intents include: – Query groups: Navigational/Informational/Transactional – Topical categorization • Manual (ODP topics) • Automatic (LDA topics) • Query clustering for non pre-defined intents Query Intents in IR
  16. 16. Pre-defined query groups: – Navigational – Informational – Transactional Query Intents in IR [Broder, SIGIR Forum 2002; Jansen et al., IPM 2008]
  17. 17. – Query terms • Navigational: Company/business/org/people • Informational: ways to/how to – Click entropy • Navigational clicks: low entropy – URL domains clicked • Navigational: usually primary domain (facebook.com) – Query length • Informational: longer queries (>2 words) – Post-click user behavior • Informational: longer sessions, reformulations Characteristics & features used for automatic classification:
  18. 18. Topical Categorization • Open Directory Project (ODP)
  19. 19. • No longer updated but a static copy publicly available at: http://www.dmoz.org/ • Crowd-sourced effort • 15 top level categories • 1031533 categories overall • 91810 editors • 3882684 websites • 90 languages Open Directory Project (ODP) [Sherman, ERIC’ 00]
  20. 20. • Topical categorization of user queries: – Make use of Open Directory Project (ODP) topic hierarchy – Crawl data from ODP to obtain training data – Train Logistic Regression to obtain topic classier • ODP topics widely used: – developing user models for personalization – ranking features Open Directory Project (ODP) [Sherman, ERIC’ 00]
  21. 21. Automatic Topic Identification • Many different models proposed • Most popular: Latent Dirichlet Allocation (LDA) topic models [Blei et al, JMLR ’03] • Unsupervised extraction of topics • Wide array of tools available
  22. 22. Latent Dirichlet Allocation [Blei et al, JMLR’03] • LDA is a generative probabilistic model of a corpus. The basic idea is that the documents are represented as random mixtures over latent topics, where a topic is characterized by a distribution over words.
  23. 23. For each document: (a) draw a topic distribution, θd ∼ Dirichlet(α), (b) for each word in the document: (i) Draw a specific topic zd,n ∼ multinomial(θd) (ii) Draw a word wd,n ∼ multinomial(βzd,n) Latent Dirichlet Allocation [Blei et al, JMLR’03]
  24. 24. Query Intents in IR • Query classification aims to classify queries into pre-defined intents Pre-defined intents include: – Query groups: Navigational/Informational/Transactional – Topical categorization • Manual (ODP topics) • Automatic (LDA topics) • Query clustering techniques based on: – k-nearest neighbors – Sequence clustering – Using session, ad-click and sponsored search data – Random walks on click graph – Reformulations & clicks
  25. 25. Random Walks on Click Graph [Craswell et al., SIGIR’07] • Consider the click graph: – Bipartite with different types of nodes (queries, documents) – An edge connects a query and a document if a click for that query-document pair is observed – The edge may be weighted according to the total number of clicks from all users
  26. 26. • Cjk : click counts associating node j to k • Define transition probabilities Pt+1|t (kij) from j to k: • s is the self-transition probability, which corresponds to the user favoring the current query or document Random Walks on Click Graph [Craswell et al., SIGIR’07]
  27. 27. Inferring Query Intent from Reformulations & Clicks Given an input query q, combine click d reformulation information to find likely user intents using three steps: 1. Expand – identify a set of possibly related queries to q (Recall is key here) – Use session co-occurrence 2. Filter – reduces the query neighborhood to more closely related queries, improving precision – Use 2 step Random walk on the the bipartite query-document click graph – All pairs of queries with a random walk similarity above a fixed threshold are connected 1. Cluster – use the random walk similarities to find intent clusters [Radlinski et al., WWW’10]
  28. 28. Section 2: Characterizing Tasks • Understanding Intents & Tasks – Query intents in IR – Search sessions – Sessions à Tasks • Characterizing Tasks across devices – Desktop based search • Taxonomy of browsing & querying behavior – Intelligent Assistants • Digital Assistants – Use cases – User engagement • Voice-only Assistants – Use cases – User engagement
  29. 29. Search Sessions as Tasks Online credit check Mortgage in principle Facebook House buying guide Quit smoking benefits Solicitors near me … Houses for sale Loans for house … 10:00am 10:03am 10:07am 12:30pm 17:00pm 17:02pm 17:06pm 18:15pm Session 1 Session 2 Session 3 Session 4 Session continuation detection [Jon08, Agi12] Session based task extraction [Luc11, Hua13, Wan13, Awa14, Luc13] A Session is a chronologically ordered sequence of user interactions with the search engine resulting in one or more queries – aimed at solving a single information need.
  30. 30. Session based IR Lot of research into the problem of segmenting and organizing query logs into semantically coherent structures Sessions allow us to: – Look beyond queries – Preserve semantic associations between adjacent query trails – Maintain context of user activity • Sessions assumed to be corresponding to model atomic search tasks
  31. 31. Session based IR Session for understanding user needs: – Query context understanding: • Prior queries in the session help lessen query ambiguity – Search result re-ranking • Session contexts helps in re-ranking results, and showing better related search suggestions – Personalization models • Session level personalization often outperforms long term user models – User satisfaction • Understanding session level user needs help in developing appropriate metrics for gauging user satisfaction
  32. 32. Section 2: Characterizing Tasks • Understanding Intents & Tasks – Query intents in IR – Search sessions – Sessions à Tasks • Characterizing Tasks across devices – Desktop based search • Taxonomy of browsing & querying behavior – Assistants • Digital Assistants – Use cases – User engagement • Voice-only Assistants – Use cases – User engagement
  33. 33. Sessions vs. Tasks 0" 10" 20" 30" 40" 50" 60" No#of#Users#(In#%)# Avg#Number#of#Tasks#Performed#in#a#Session# 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 10+" • Sessions don’t represent atomic information needs • Users often multi-task in a session tϕ = 30 minutes An impression finding flight ticketsbooking hotels
  34. 34. From Sessions to Tasks • Sessions should NOT be the focal units in IR • More than 70% of sessions are multi-task – Users differ based on their multi-tasking habits – Some topics prone to multi-tasking than others • 75% of search tasks tend to span across multiple sessions [Jones and Klinkner, CIKM '08]
  35. 35. What is a Task? • A search task is an atomic information need resulting in one or more queries [Jones and Klinkner, CIKM '08] • Complex search task: A set of related information needs, resulting in one or more (possibly complex) tasks. Credit check House buying guide … Houses for sale Loans for house 17:00pm 17:02pm 17:06pm 18:15pm Session 1 Session 2 Improve credit score 18:25pm
  36. 36. Rishabh takes over
  37. 37. Section 2: Characterizing Tasks • Understanding Intents & Tasks – Query intents in IR – Search sessions – Sessions à Tasks • Characterizing Tasks across devices – Desktop based search • Taxonomy of browsing & querying behavior – Intelligent Assistants • Digital Assistants – Use cases – User engagement • Voice-only Assistants – Use cases – User engagement
  38. 38. Taxonomy of Tasks [Broder, SIGIR Forum 2002; Russell et al., HICSS 2009; Bailey et al., NII Shannon 2012]
  39. 39. Typical Web Tasks by Session • Definition: All web activities including browsing behavior and search behavior [Bailey et al., NII Shannon 2012]
  40. 40. Top Search Tasks by Session [Bailey et al., NII Shannon 2012]
  41. 41. Complex Tasks and Sub-tasks [Bailey et al., NII Shannon 2012] Comparison Task broken down into the following sub-tasks: • Explore dimensions for comparison (size, color, capacity, megapixel) • Compile and refine a list of choices (comparable models) • Find details about a choice • Read reviews about a choice • Read side by side comparisons • Act on a comparison decision
  42. 42. Section 2: Characterizing Tasks • Understanding Intents & Tasks – Query intents in IR – Search sessions – Sessions à Tasks • Characterizing Tasks across devices – Desktop based search • Taxonomy of browsing & querying behavior – Intelligent Assistants • Digital Assistants – Use cases – User engagement • Voice-only Assistants – Use cases – User engagement
  43. 43. Digital Assistants: Use Cases • Different from traditional web search – Verbal & textual interactions – Plethora of tasks (files, reminders, surf the web, etc) • Motivates the need to develop better understanding of user interactions – Better user interest models – Help in task support & completion – Develop metrics & gauge satisfaction – Envision future use case scenarios
  44. 44. Investigating use cases of a Digital Assistant
  45. 45. Session Characteristics • Analysis by session: Over 80% sessions have 1-5 queries – Similar to traditional web search • Analysis by traffic: longer sessions bring in over 50% of traffic – Traditional IR: longer sessions ~ exploring or struggling [Odijk et al. CIKM’15] – Open Question: Is that the case here too? 0 0.1 0.2 0.3 0.4 0.5 0.6 1 "2-5" "6-10" "11-20" 21-50 51+ % Sessions 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 1 "2-5" "6-10" "11-20" 21-50 51+ % IMPRESSIONS
  46. 46. Input Modality • Desktop search is primarily text based • Interaction with intelligent assistants is mostly speech based • Longer sessions are mostly speech driven • Speech queries are much longer 1.8 1.85 1.9 1.95 2 2.05 2.1 Speech Text Avg No of Words 9.6 9.8 10 10.2 10.4 10.6 10.8 11 11.2 11.4 11.6 11.8 Speech Text Avg No of Characters
  47. 47. Common Use-Case Scenarios • General search • Commands – Alarms, camera, system settings • Answers – Find instant answers • MyStuff – Search local files & folders 0 0.1 0.2 0.3 0.4 0.5 MyStuff Answers Commands GeneralSearch Percentage Interactions Cortana Desktop Use-cases [Mehrotra et al., CAIR 2017]
  48. 48. Common Use-Case Scenarios Spread across session length • MyStuff – Over 70% single query sessions originate from MyStuff – Proportion on MyStuff steadily decreases as the session length increases • Instant Answers – Increasing proportion of sessions with session length • Steady proportions of General Search & Commands 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Use-case Mix-Up in Sessions MyStuff Answers CommandControl GeneralSearch [Mehrotra et al., CAIR 2017]
  49. 49. Beyond Traditional IR Commands Answers 0 0.1 0.2 0.3 0.4 0.5 Open App Weather How To BilingualDict Math News Answers Time zone Entity Lookup Percentage Interactions Beyond Chitchat & General Search • Top Commands: reminders, alarms, music • Users seek direct answers: weather, HowTos, Math etc • Envision appropriate changes as system evolves [Mehrotra et al., CAIR 2017]
  50. 50. User Interaction Signals Traditional CortanaDesktop Has Scroll Traditional CortanaDesktop Scroll Distance Traditional CortanaDesktop Pointer Distance Traditional CortanaDesktop Hover Count Click Count SCC Click based differences CortanaDesktop Traditional Traditional CortanaDesktop Query DT
  51. 51. User Interaction with Proactive Cards • Similar trends to search queries: – More clicks at the top cards – Cards at lower ranks get relatively less engagement [Shokouhi and Guo, SIGIR 2015]
  52. 52. User Interaction with Proactive Cards • Temporal trends – Sports: weekend / Traffic: weekdays [Shokouhi and Guo, SIGIR 2015]
  53. 53. Desktop vs Mobile vs Proactive [Shokouhi and Guo, SIGIR 2015]
  54. 54. Section 2: Characterizing Tasks • Understanding Intents & Tasks – Query intents in IR – Search sessions – Sessions à Tasks • Characterizing Tasks – Desktop based search • Taxonomy of browsing & querying behavior – Digital Assistants • Use cases • User engagement – Voice-only Assistants • Use cases • User engagement
  55. 55. Voice-only Assistants • No textual input • More conversational in nature
  56. 56. Tasks people do on Voice-only Assistants Source: comScore
  57. 57. Where do people use the Smart Speakers? *Other devices may not be as easily accessible with occupied hands. Source: NPR, Edison Research
  58. 58. Summary: Characterizing Tasks • Understanding User Intents & Tasks – Different abstractions of understanding user needs: Queries à Sessions à Tasks – Methods for intent identification • Classification • Clustering – Search sessions and Tasks • Multitasking sessions • Tasks provide a better abstraction • Characterizing Tasks – Taxonomy of web search tasks – Traditional desktop based search tasks are different than emerging tasks on smart assistants – User interactions differ across different devices – More contextual information available for smart assistants
  59. 59. Outline of the Tutorial • Section 1: Introduction • Section 2: Characterizing Tasks • Section 3: Tasks Extraction Algorithms • Section 4: Task based Evaluation • Section 5: Applications

×