A model of  collaborative search  Gene Golovchinsky and  Jeremy Pickens FX Palo Alto Laboratory, Inc.
The elephant in the room <ul><li>Many studies have show that info seeking is not a solitary pursuit </li></ul><ul><ul><li>...
What about Social Search? <ul><li>Social feedback </li></ul><ul><ul><li>Observe patterns of behavior or opinion </li></ul>...
But… <ul><li>Similarity of query terms does not imply similarity of info need </li></ul><ul><ul><li>Inference is imperfect...
Collaborative Search vs.  Social Search <ul><li>Social (feedback) search: </li></ul><ul><ul><li>Infers  similar informatio...
A history of modeling <ul><li>Single-user information seeking </li></ul><ul><ul><li>Iterative process (The usual suspects)...
A model for collaborative search <ul><li>User behavior </li></ul><ul><ul><li>Intent </li></ul></ul><ul><ul><ul><li>Implici...
Intent <ul><li>Implicit </li></ul><ul><ul><li>User may be aware that others’ data are used to inform search </li></ul></ul...
Depth of mediation <ul><li>Communication </li></ul><ul><ul><li>People communicate about search tasks, about search results...
Data synchronization <ul><li>Symmetric influence </li></ul><ul><ul><li>Data generated by one person are available to  all ...
Examples <ul><li>Ariadne (Twidale, Nichols, Paice, 1997) </li></ul><ul><ul><li>Explicit intent, communication-only mediati...
Example: Ariadne  <ul><li>Explicit intent </li></ul><ul><ul><li>People are working toward a shared information need </li><...
Example: Recommendation Systems  and Social Search <ul><li>Implicit intent </li></ul><ul><ul><li>Each person is pursuing a...
Example: SearchTogether <ul><li>Explicit intent </li></ul><ul><ul><li>People are working toward a shared information need ...
Example: Cerchiamo <ul><li>Explicit intent </li></ul><ul><ul><li>People are working toward a shared information need </li>...
Cerchiamo, some details… <ul><li>Two users in different roles </li></ul><ul><ul><li>Prospector opens up new paths for expl...
Algorithmic Mediation Engine Input Coordinator User 1 Output Coordinator User 2 User Interface Regulator (Roles) Collabora...
What’s the purpose of this complexity? <ul><li>Collaboration seems to help on difficult search tasks </li></ul><ul><ul><li...
Efficiency:  Large advantage for sparse topics <ul><li>Average # examined @15 minutes </li></ul><ul><li>Plentiful topics <...
Uniqueness System A System B
Uniqueness All Other Systems System A System B
Effectiveness: Uniqueness
Summary of results <ul><li>For difficult information needs </li></ul><ul><ul><li>Collaboration improves efficiency </li></...
Comments on Evaluation <ul><li>Need to invent new ways to assess performance </li></ul><ul><ul><li>Dividing by number of s...
References <ul><li>Ackerman, M. S. and McDonald, D. W. (1996) Answer Garden 2: merging organizational memory with collabor...
Questions? CFP: 2 nd  International Workshop on  Collaborative Information Seeking  Held in conjunction with  CSCW 2010 in...
Upcoming SlideShare
Loading in …5
×

A Model Of Collaborative Search

3,049 views

Published on

In the library sciences, information seeking has long been recognized as a collaborative activity, and recent work has attempted to model group information seeking behavior. Until recently, technological support for group-based information seeking has been limited to collaborative filtering and "social search" applications. In the past two years, however, a new kind of technologically-mediated collaborative search has been demonstrated in systems such as SearchTogether and Cerchiamo. This approach is more closely grounded in the library science interpretation of collaboration: rather than inferring commonality of interest through similarity of queries (social search), the new approach assumes an explicitly-shared information need for a group. This allows the system to focus on mediating the collaboration rather than detecting its presence. In this talk, we describe a model that captures both user behavior and system architecture, describe its relationship to other models of information seeking, and use it to classify existing multi-user search systems. We also describe implications this model has for design and evaluation of new collaborative information seeking systems.

Talk presented at NIST on October 22, 2009.

Published in: Technology, Education

A Model Of Collaborative Search

  1. 1. A model of collaborative search Gene Golovchinsky and Jeremy Pickens FX Palo Alto Laboratory, Inc.
  2. 2. The elephant in the room <ul><li>Many studies have show that info seeking is not a solitary pursuit </li></ul><ul><ul><li>Allen (1977) </li></ul></ul><ul><ul><li>O’Day and Jeffries (1993) </li></ul></ul><ul><ul><li>Twidale, Nichols and Paice (1997) </li></ul></ul><ul><ul><li>Talja (2002) </li></ul></ul><ul><ul><li>Hansen & J ä rvelin (2005) </li></ul></ul><ul><ul><li>Hyldeg å rd (2006) </li></ul></ul><ul><ul><li>Hertzum (2008) </li></ul></ul><ul><ul><li>Morris (2008) </li></ul></ul><ul><ul><li>Evans and Chi (2008) </li></ul></ul><ul><ul><li>Reddy & Jansen (2008) </li></ul></ul><ul><li>Many tasks have been shown to benefit from collaboration </li></ul><ul><ul><li>Engineers collaborating on design </li></ul></ul><ul><ul><li>Reference librarians working with patrons </li></ul></ul><ul><ul><li>Medical / drug research teams </li></ul></ul><ul><ul><li>Student project teams </li></ul></ul><ul><ul><li>Patent Search (multiple legal clerks working on same case) </li></ul></ul><ul><ul><li>Patient care teams (multiple doctors with differing expertise caring for the same individual) </li></ul></ul><ul><ul><li>Families making important decisions (e.g. real estate, travel) </li></ul></ul><ul><ul><li>Friends or colleagues planning activities together </li></ul></ul>Yet most systems for info seeking are designed for only one person
  3. 3. What about Social Search? <ul><li>Social feedback </li></ul><ul><ul><li>Observe patterns of behavior or opinion </li></ul></ul><ul><ul><li>Identify useful documents </li></ul></ul><ul><ul><li>Infer similar information needs </li></ul></ul><ul><ul><li>Retrieve useful documents </li></ul></ul><ul><li>Social answering </li></ul><ul><ul><li>Explicit use of social networks to help answer questions </li></ul></ul><ul><ul><ul><li>Answer Garden (Ackerman and McDonald, 1996) </li></ul></ul></ul><ul><ul><ul><li>Aardvark (www.vark.com) </li></ul></ul></ul><ul><ul><ul><li>Ad hoc: uses of Facebook, Twitter, other comm. channels to help find information (Evans and Chi, 2008) </li></ul></ul></ul><ul><li>Taxonomy proposed by Chi (2009) </li></ul>
  4. 4. But… <ul><li>Similarity of query terms does not imply similarity of info need </li></ul><ul><ul><li>Inference is imperfect; users may have differing needs </li></ul></ul><ul><ul><li>Indirect and noisy feedback </li></ul></ul><ul><li>Finding same documents as others may not be useful in some cases </li></ul><ul><ul><li>Known-item search as the consumption of knowledge vs. </li></ul></ul><ul><ul><li>Exploratory search as the creation of knowledge </li></ul></ul><ul><li>Interaction with system </li></ul><ul><ul><li>Is still often solitary </li></ul></ul><ul><ul><li>Is limited to communication with others </li></ul></ul><ul><li>Social search is not truly collaborative </li></ul>
  5. 5. Collaborative Search vs. Social Search <ul><li>Social (feedback) search: </li></ul><ul><ul><li>Infers similar information need </li></ul></ul><ul><ul><li>Leverages wisdom of crowds for known-item search </li></ul></ul><ul><li>Collaborative search: </li></ul><ul><ul><li>Assumes shared information need </li></ul></ul><ul><ul><li>Combines multiple perspectives to improve search results </li></ul></ul>
  6. 6. A history of modeling <ul><li>Single-user information seeking </li></ul><ul><ul><li>Iterative process (The usual suspects) </li></ul></ul><ul><li>Collaboration on teams </li></ul><ul><ul><li>Group cohesion (Hertzum, 2007) </li></ul></ul><ul><ul><li>Classes of collaborative activity (J ä rvelin and Hansen, 2005) </li></ul></ul><ul><li>Social answering </li></ul><ul><ul><li>Patterns of information flow among people (Evans and Chi, 2008) </li></ul></ul><ul><li>What’s missing? </li></ul><ul><ul><li>Interaction, coordination, mechanics </li></ul></ul><ul><ul><li>Foundation for principled design for collaboration </li></ul></ul>
  7. 7. A model for collaborative search <ul><li>User behavior </li></ul><ul><ul><li>Intent </li></ul></ul><ul><ul><ul><li>Implicit vs. explicit </li></ul></ul></ul><ul><li>System behavior </li></ul><ul><ul><li>Depth of mediation </li></ul></ul><ul><ul><ul><li>Communication vs. UI vs. algorithmic mediation </li></ul></ul></ul><ul><ul><li>Synchronization </li></ul></ul><ul><ul><ul><li>Asymmetric vs. Symmetric </li></ul></ul></ul>
  8. 8. Intent <ul><li>Implicit </li></ul><ul><ul><li>User may be aware that others’ data are used to inform search </li></ul></ul><ul><ul><li>System infers similarity of information need </li></ul></ul><ul><ul><li>System recommends documents based on inference of similarity </li></ul></ul><ul><ul><li>Good for finding what others have already found, thus </li></ul></ul><ul><ul><ul><li>May not effective for exploratory search </li></ul></ul></ul><ul><li>Explicit </li></ul><ul><ul><li>Users explicitly declare shared information need </li></ul></ul><ul><ul><li>System combines contributions from collaborators to find new information </li></ul></ul><ul><ul><li>Good for exploratory search </li></ul></ul>
  9. 9. Depth of mediation <ul><li>Communication </li></ul><ul><ul><li>People communicate about search tasks, about search results </li></ul></ul><ul><ul><li>Neither the interface nor the algorithms know that multiple people are involved </li></ul></ul><ul><li>UI </li></ul><ul><ul><li>Each person uses system independently </li></ul></ul><ul><ul><li>Retrieval system is unaware of multiple people </li></ul></ul><ul><li>Algorithmic </li></ul><ul><ul><li>Each person’s contributions are tracked separately by the retrieval system </li></ul></ul><ul><ul><li>Contributions may be combined to produce desired retrieval effects </li></ul></ul><ul><li>Aspects are cumulative </li></ul><ul><ul><li>UI may also include communication </li></ul></ul><ul><ul><li>Algorithmic mediation may also include UI and communication </li></ul></ul>
  10. 10. Data synchronization <ul><li>Symmetric influence </li></ul><ul><ul><li>Data generated by one person are available to all collaborators for the same search task </li></ul></ul><ul><ul><ul><li>SearchTogether </li></ul></ul></ul><ul><ul><ul><li>Cerchiamo </li></ul></ul></ul><ul><li>Asymmetric influence </li></ul><ul><ul><li>Some people do not see contributions of others </li></ul></ul><ul><ul><ul><li>Recommendations </li></ul></ul></ul><ul><li>Synchronization is not synchronicity </li></ul><ul><ul><li>No implication when people search </li></ul></ul><ul><ul><li>No requirement of WUSIWIS </li></ul></ul><ul><ul><li>Describes availability of other peoples’ data with respect to an information need </li></ul></ul>
  11. 11. Examples <ul><li>Ariadne (Twidale, Nichols, Paice, 1997) </li></ul><ul><ul><li>Explicit intent, communication-only mediation, no data synchronization </li></ul></ul><ul><li>Recommendation systems (e.g., Linden et al, 2003) </li></ul><ul><ul><li>Implicit intent, algorithmic mediation, asymmetric data synchronization </li></ul></ul><ul><li>SearchTogether (Morris,& Horvitz, 2007) </li></ul><ul><ul><li>Explicit intent, UI mediation, symmetric data synchronization </li></ul></ul><ul><li>Cerchiamo (Pickens, et al., 2008) </li></ul><ul><ul><li>Explicit intent, algorithmic mediation, symmetric data synchronization </li></ul></ul>
  12. 12. Example: Ariadne <ul><li>Explicit intent </li></ul><ul><ul><li>People are working toward a shared information need </li></ul></ul><ul><li>Communication-level mediation </li></ul><ul><ul><li>Chat about search </li></ul></ul><ul><ul><li>System does not mediate </li></ul></ul><ul><li>No data synchronization </li></ul><ul><ul><li>Each person is responsible for individual search </li></ul></ul><ul><li>Twidale, Nichols and Paice (1997) </li></ul>
  13. 13. Example: Recommendation Systems and Social Search <ul><li>Implicit intent </li></ul><ul><ul><li>Each person is pursuing an individual agenda </li></ul></ul><ul><li>Algorithmic mediation </li></ul><ul><ul><li>System keeps track of each user’s actions </li></ul></ul><ul><ul><li>System aggregates information to generate recommendations </li></ul></ul><ul><li>Asymmetric data synchronization </li></ul><ul><ul><li>Only “future” searchers benefit from recommendations </li></ul></ul><ul><li>Many implementations </li></ul><ul><ul><li>Linden et al (2003) </li></ul></ul><ul><ul><li>Smyth et al (2004) </li></ul></ul>
  14. 14. Example: SearchTogether <ul><li>Explicit intent </li></ul><ul><ul><li>People are working toward a shared information need </li></ul></ul><ul><li>UI-level mediation </li></ul><ul><ul><li>System keeps track of each user’s documents, judgments </li></ul></ul><ul><ul><li>System does not differentiate among searchers with respect to retrieval </li></ul></ul><ul><li>Symmetric data synchronization </li></ul><ul><ul><li>Judgments of relevance are available to all collaborators </li></ul></ul><ul><li>Morris and Horvitz (2007) </li></ul>
  15. 15. Example: Cerchiamo <ul><li>Explicit intent </li></ul><ul><ul><li>People are working toward a shared information need </li></ul></ul><ul><li>Algorithmic mediation </li></ul><ul><ul><li>System keeps track of each user’s documents, judgments </li></ul></ul><ul><ul><li>System integrates inputs from both searchers to produce search results, term suggestions </li></ul></ul><ul><li>Symmetric data synchronization </li></ul><ul><ul><li>Terms and documents are a shared among all searchers </li></ul></ul><ul><ul><li>Different roles are assigned different interfaces </li></ul></ul><ul><li>Pickens, Golovchinsky, Shah, Qvarfordt, and Back (2008) </li></ul>
  16. 16. Cerchiamo, some details… <ul><li>Two users in different roles </li></ul><ul><ul><li>Prospector opens up new paths for exploration </li></ul></ul><ul><ul><li>Miner digs through promising results </li></ul></ul><ul><li>Bi-directional influence </li></ul><ul><ul><li>Prospector issues queries </li></ul></ul><ul><ul><li>System shows shots for miner to judge </li></ul></ul><ul><ul><li>Miner makes relevance judgments </li></ul></ul><ul><ul><li>System suggests search terms to prospector </li></ul></ul><ul><li>System coordinates </li></ul><ul><ul><li>Different inputs from each person </li></ul></ul><ul><ul><li>Different outputs to each person </li></ul></ul>
  17. 17. Algorithmic Mediation Engine Input Coordinator User 1 Output Coordinator User 2 User Interface Regulator (Roles) Collaborative Search
  18. 18. What’s the purpose of this complexity? <ul><li>Collaboration seems to help on difficult search tasks </li></ul><ul><ul><li>TRECVid 2007 ad hoc task </li></ul></ul><ul><li>Cerchiamo vs. post-hoc pooling of results from two “prospector” users </li></ul><ul><ul><li>24 searchers </li></ul></ul><ul><ul><li>Paired standalone vs. paired collaborative </li></ul></ul><ul><li>Metric </li></ul><ul><ul><li>Viewed precision = Fraction of viewed documents that is relevant </li></ul></ul><ul><li>Results </li></ul><ul><ul><li>When finding documents is easy (plentiful topics) </li></ul></ul><ul><ul><ul><li>No advantage to collaboration </li></ul></ul></ul><ul><ul><li>When finding documents is harder (sparse topics) </li></ul></ul><ul><ul><ul><li>Collaboration is more efficient (more relevant shots found despite examining fewer documents) </li></ul></ul></ul>
  19. 19. Efficiency: Large advantage for sparse topics <ul><li>Average # examined @15 minutes </li></ul><ul><li>Plentiful topics </li></ul><ul><ul><li>Collaborative: 2,352 </li></ul></ul><ul><ul><li>Post hoc Merged: 2,168 </li></ul></ul><ul><li>Sparse topics </li></ul><ul><ul><li>Collaborative: 2,877 </li></ul></ul><ul><ul><li>Post hoc Merged: 3,787 </li></ul></ul>
  20. 20. Uniqueness System A System B
  21. 21. Uniqueness All Other Systems System A System B
  22. 22. Effectiveness: Uniqueness
  23. 23. Summary of results <ul><li>For difficult information needs </li></ul><ul><ul><li>Collaboration improves efficiency </li></ul></ul><ul><ul><li>Collaboration improves effectiveness </li></ul></ul><ul><li>Preliminary evidence </li></ul><ul><ul><li>Need to perform more evaluations </li></ul></ul><ul><ul><li>Need to evaluate on text-only documents </li></ul></ul><ul><li>Conception of the task affects design </li></ul><ul><ul><li>Social search improves known-item search </li></ul></ul><ul><ul><li>Collaboration improves exploratory search </li></ul></ul>
  24. 24. Comments on Evaluation <ul><li>Need to invent new ways to assess performance </li></ul><ul><ul><li>Dividing by number of searchers not always appropriate </li></ul></ul><ul><ul><li>Mythical “man month” </li></ul></ul><ul><li>Need outcome-centered measures </li></ul><ul><ul><li>Interactive task </li></ul></ul><ul><ul><li>MAP not appropriate </li></ul></ul><ul><ul><li>Set-based metrics more appropriate </li></ul></ul><ul><ul><li>“ Viewed” recall and precision capture human judgment, not just system performance </li></ul></ul><ul><li>How to compare iterative processes? </li></ul><ul><ul><li>Document-centric rather than query-centric metrics? </li></ul></ul><ul><ul><li>Need to account for relevant-but-redundant documents </li></ul></ul>
  25. 25. References <ul><li>Ackerman, M. S. and McDonald, D. W. (1996) Answer Garden 2: merging organizational memory with collaborative help. In Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work (Boston, Massachusetts, United States, November 16 - 20, 1996). M. S. Ackerman, Ed. CSCW '96. ACM, New York, NY, 97-105. </li></ul><ul><li>Allen, T. (1977) Managing the flow of technology: Technology transfer and the dissemination of technological information within the R&D organization. MIT Press </li></ul><ul><li>Chi, Ed H. (2009) Information Seeking Can Be Social, Computer , vol. 42, no. 3, pp. 42-46, Mar. 2009 </li></ul><ul><li>Evans, B.; Chi, E. H. (2008) Towards a Model of Understanding Social Search. In Proc. of Computer-Supported Cooperative Work (CSCW). ACM Press. San Diego, CA  </li></ul><ul><li>Hansen, P. and Järvelin, K. (2005) Collaborative Information Retrieval in an information-intensive domain. Information Processing&Management 41. pp. 1101–1119. </li></ul><ul><li>Hertzum (2008) Collaborative information seeking: The combined activity of information seeking and collaborative grounding. Information Processing&Management 44 pp. 957–962. </li></ul><ul><li>Hyldeg å rd, J. (2006) Collaborative information behaviour––exploring Kuhlthau’s Information Search Process model in a group-based educational setting. Information Processing&Management 42. pp. 276–298 </li></ul><ul><li>Linden, G., Smith, B. and York, J. (2003) Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Computing , 7 (1). 76-80.  </li></ul><ul><li>Morris, M. R. and Horvitz, E. (2007) SearchTogether: an interface for collaborative web search. In Proceedings of UIST '07 . ACM, New York, NY, 3-12. </li></ul><ul><li>O'Day, V. & Jeffries, R. (1993) Information artisans: patterns of result sharing by information searchers. In: Proceedings of the ACM Conference on Organizational Computing Systems, 98-107. ACM Press.   </li></ul><ul><li>Pickens, J., Golovchinsky, G., Shah, C., Qvarfordt, P., and Back, M. (2008) Algorithmic Mediation for Collaborative Exploratory Search. In Proceedings of SIGIR 2008 , July 22-25. ACM Press.  </li></ul><ul><li>Reddy & Jansen (2008) Learning about Potential Users of Collaborative Information Retrieval Systems. In Proc. JCDL Collab IR Workshop </li></ul><ul><li>Smyth, B., Balfe, E., Freyne, J., Briggs, P., Coyle, M. and Boydell, O. (2004) Exploiting Query Repetition and Regularity in an Adaptive Community-Based Web Search Engine. UMUAI 14, 5. 383-423. </li></ul><ul><li>Talja, S. (2002) Information sharing in academic communities: types and levels of collaboration in information seeking and use. New Review of Information Behavior Research , 3(1) , 143-159. </li></ul><ul><li>Twidale, M., Nichols, D. & Paice, C. (1997) Browsing is a collaborative process. Information Processing and Management , 33(6) , 761-783. </li></ul>
  26. 26. Questions? CFP: 2 nd International Workshop on Collaborative Information Seeking Held in conjunction with CSCW 2010 in Savannah, GA February 2010 http://www.fxpal.com/ cscw2010cis

×