Contextual support for collaborative information retrieval
•Download as PPTX, PDF•
1 like•426 views
Recent research shows that Collaborative Information Retrieval (CIR), in which two or more users collaborate on the same search task, has become increasingly popular. The presence of both search and collaboration behaviors makes CIR a complex search format, which further drives a critical need to understand CIR's search context. The contextual support for CIR should consider search contexts derived from both team members' search histories (including users' own search histories and partners' search histories) and their explicit collaboration (e.g., chatting). As it stands, existing studies on contextual search support only focus on Individual Information Retrieval (IIR) and only utilize individuals' own search histories. In this paper, we examine the unique search contexts (e.g., partners' search histories and team collaboration histories) in CIR. Based on a user study data collection with 54 participants, we find that compared to the use of individuals' own search histories, CIR contextual support is more effective when utilizing partners' search histories and teams' collaboration behaviors. More interestingly, though the explicit communication information (i.e., chat content) often involves massive noisy information, involving such noise does not affect the ranking of relevant documents since it also does not appear in relevant documents.
Recommended
More Related Content
Similar to Contextual support for collaborative information retrieval
Similar to Contextual support for collaborative information retrieval (20)
Recently uploaded
Recently uploaded (20)
Contextual support for collaborative information retrieval
- 1. Contextual Support for Collaborative Information Retrieval Shuguang Han, Daqing He, Zhen Yue and Jiepu Jiang 1 Shuguang Han, Daqing He, Zhen Yue, and Jiepu Jiang. 2016. Contextual Support for Collaborative Information Retrieval. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR '16). ACM, New York, NY, USA, 33-42. DOI=http://dx.doi.org/10.1145/2854946.2854963
- 2. Motivation Collaborative Information Retrieval (COL) VS. Individual Information Retrieval (IND) Studies observed an increasing trend of COL, in which people search together for the same task COL provides unique & interesting contextual factors for search support team project ski trip students family Communication & Coordination Shared Search History Search Search 2
- 3. Research Questions RQ1: Are IND contextual support directly applicable in COL without handling complex COL interactions? 3 RQ2: Can partners’ search histories be employed for contextual search support in COL? RQ3: Can team members’ chat content be applied for contextual search support in COL?
- 4. Contextual Support for COL Search contexts Five types (since chat is highly interactive and is not isolated, we merge self- and partner-chat) 4 Contextual search support models Document re-ranking based on search contexts Experiment setup for contextual search support Data about user search contexts Ground-truth
- 5. Obtaining Data Collection User study A self-developed collaborative web search system Search Shared Workspace Shared Query History Communication & Coordination 5
- 6. Obtaining Data Collection Search Tasks T1: Academic Task Writing a report to summarize the development of SNS T2: Leisure Task Planning a trip to Helsinki User Study Procedure Conditions: Collaborative search (COL) & Individual search (IND) Within subject design Each team (in COL) / individual(in IND) works 30 mins for each task Post-task questionnaire for relevance rating of each saved document User Study Data 18 pairs (for COL) and 18 individuals (for IND) 970 queries, 1,384 clicks and 909 saved documents 6
- 7. Ground-truth ● For each team {U, V}, pooling document relevance ○ Aggregating document relevance by averaging relevance judgement from other users ● Personalize ground-truth for each query q ○ Pooled relevant documents already-saved relevant documents 7
- 8. Utilizing Self Search Histories in COL RQ1: Are IND contextual support directly applicable in COL? 8 NDCG@NMAP G: pure google results (baseline) G + H_QUS: re-rank Google results using self query history G + H_CLS: re-rank Google results using self click history Take-away messages ● Self search history is also useful in COL ● QU > CL (different from [Shen et al. 2004]) ○ may because of the task nature ○ may because of that self search behaviors in COL are influenced by collaboration
- 9. Utilizing Self Search Histories in IND Academic Leisure #chat messages 21.56 (20.98) 38.86 (27.24) #queries 12.97 (6.98) 9.25 (5.30) #click-through 24.00 (13.23) 14.44 (7.25) search-intensive task collaboration-intensive task 9 NDCG@NMAP Take-away messages ● IND is consistent with COL for Academic task ● IND is inconsistent with COL for Leisure task ○ Hypothesis: Strong collaboration effect in leisure task G: pure google results (baseline) G + H_QUS: re-rank Google results using self query history G + H_CLS: re-rank Google results using self click history
- 10. Utilizing Partners’ Search Histories in COLNDCG@N RQ2: Can partners’ search histories be employed for contextual search support in COL? MAP 10 G + H_QU: re-rank Google by both (self + partner) query history G + H_CL: re-rank Google by both (self + partner) click history Take-away messages ● Incorporating search histories from partners can provide a better contextual support
- 11. Utilizing Chat Histories in COL Take-away messages ● Chat-based approach performs the best in the leisure task ● Chat-based approach outperforms click-based method in the academic task ● This finding goes beyond our expectation ○ Chat contains a lot noise (our previous study find that 30% of chats are irrlevant to tasks) NDCG@N RQ3: Can team members’ chat content be applied for contextual search support in COL? G + H_CH: re-rank Google results by chat history MAP 11
- 12. Category Description Task social (TS) Chats concerning attitude to obtained resources Task coordination (TC) Chats about the coordination of a task Task content (TT) Chats related to the content of the search task Non-task (NT) Chats that are not related to the search task G + H_X: re-rank Google with {TS, TC, TT, NT} chat history NDCG@NMAP Take-away messages ● Any type of chat (even for NT) helps improve search ● Involving all chat messages achieves the best Categorization schema for manual chat message coding [2] 12 Utilizing Chat Histories in COL
- 13. Analyzing Non-Task Chat in COL Task NT (×10-3) Others (×10-3) T1 p(X|C) 0.520 1.936 p(X|R) 0.509 2.422 T2 p(X|C) 0.333 2.120 p(X|R) 0.346 3.139 Why NT works? ● They contain useful information. Each chat message refers to all the content a user typed in chat box before she hits the submit button, which may cover multiple types of chat content. ● Noise in NT (e.g., lol, haha, okay) does not hurt result ranking 13 ● p(NT|R) ≈ p(NT|C) ● p(Others|R) > p(Others|C) p(X|C) p(X|R) corpus relevant docs C R
- 14. Take-away Messages Contextual support method developed in IND can be directly applied in COL Task may affect the utility of different search contexts Partners’ search histories can help for contextual support in COL Combination of self and partners’ search histories provides better utility Chat messages can help in for contextual support in COL Noise in chat messages may not affect the utility of chat-based context 14
- 15. Q & A 15
- 16. Task Social well, hello there yeah! we are going to Helsinki! everything looks great so far! Task Coordination you do stats and I’ll do impacts on students and professionals have you done impact yet? Task Content ok so outdoor activities will be hard 16 Examples of Different Chat Types
Editor's Notes
- http://www.pitt.edu/~shh69/resources/ccir-preprint.pdf
- Recent studies have observed an increasing trend of COL, in which two or more people search together for the same search task. For example, a group of students may search and collaborate together on a course project and family members may search and plan together for a ski trip. Comparing to IND that only contains search activities, COL has several unique characteristics such as communication, coordination and shared search history. These are unique & interesting factors that could be useful for better contextual search support.
- To understand the usefulness of each type of search context, we study the following research questions. Suppose that we have two users working on a same search task. To support query q_{2n} for u_2, we can follow the method developed in IND that utilizes u_{2}’s self query and click histories. Although they have been demonstrated to be useful in IND, it remains unclear in COL since users’ search behaviors are affected Note that communication and collaboration will affect search behaviors of COL. This leads to RQ1. Besides self-search history, we can also obtain partners’ search history. This is a unique contextual information and we want to examine its usefulness in RQ2. Additionally, explicit communication, in this case chat, is another unique search context and we want to understand its effectiveness in RQ3.
- We mentioned several times about search contexts and contextual support. Here, we provide a detailed explanation. In this paper, we care about search contexts inferred from search history and chat history, where search history can be self and partner. Then, each type of search context will be used in contextual search support. In this study, we examine how each does search context support find relevant documents, in which we apply the document re-ranking approach. Then, we evaluate the effectiveness of each search context based on MAP and nDCG re-ranked search results. However, this requires the ground-truth of relevant documents.
- Since there lacks publicly-available data collections & collaborative web search systems. Our data collection is obtained through a user study based on a self-developed CIR system. Here, we show an interface to illustrate the major system functionalities. Users can search a query and system will, by default, return Google results. They can also chat, look at other users’ search queries and saved documents,
- Details (537 queries from T1 and 433 from T2)
- When building the ground-truth document relevance for a pair {U,V} Here, ground-truth is changed with the change of different teams and contexts - we only care about these new & relevant documents.
- Now, we talk about our experimental results. We firstly examine RQ1, in which we try to understand whether query and click history-based search contexts often applied in IND can also be used in COL. Note that users’ search behaviors will be different since collaboration can interrupt individuals’ search process. The MAP and nDCG evaluations are provided in the left diagram, where the gray means the performance for pure google-based ranking, red indicates re-ranking based on query history and blue means click history. We find that same as IND, self search history is also useful in COL. However, different from IND, QU > CL in COL which may due to: (1) the task nature, meaning that QU will also be better than CL even if in IND; (2) it may also due to that individual’s search behaviors will be affected by collaboration
- To test the above two reasons, we apply the same procedure as we did for COL but examine the effectiveness of self search history for IND. Again, the results are shown in this diagram, where we find that (1) QU > CL in academic task which is consistent with COL; (2) while CL has no difference with QU and seems to be even better. This is inconsistent with COL and we think it is because of the strong collaboration effect for leisure task.
- Red one uses BOTH query histories from self + partner, Blue one uses BOTH click histories from self + partner. Pink and light blue ones are self-based query and click histories. We find that including both search histories are useful.
- not every chat seems relevant to the task. we want to analyze different types of chat.. This motivates us to look further for the task
- find a good example for different chat types. Back up slides
- too much time. even pure noise, doesn’t matter reformulate the presentation of table & figure. Boundary.
- The whole conclusions & future work can be found in our paper. Here are two take-away messages we want to highlight. (Two or session -- chat & partners)
- pre-search knowledge