CHI2009 MrTaggy Tag-based Search Browser Intro and Evaluation

Information Foraging: Tuesday, 9:00 AM - 10:30 AM An Elementary Social Information Foraging Model Peter Pirolli Remembrance of Things Tagged: How Tagging Effort Affects Tag Production and Human Signpost from the Masses: Memory Raluca Budiu, Peter Pirolli, Lichan Hong Signpost from the Masses: Learning Effects in an Exploratory Social Tag Search Browser Learning Effects in an Exploratory Yvonne Kammerer, Rowan Nairn, Peter Pirolli, Ed H. Chi StudyingSocial Tag Search Browser Wikipedia: Wednesday, 11:30 AM - 1:00 PM So You Know You’re Getting the Best Possible Information: A Tool that Increases Wikipedia Credibility Peter Pirolli, Evelin Wollny, Bongwon Rowan Nairn, Peter Pirolli, Ed H. Chi Yvonne Kammerer*, Suh What's in Wikipedia? Mapping Topics and Conflict Using Socially Annotated Category Contact: Structure Ed H. Chi, Ph.D. Aniket Kittur, Ed H. Chi, Bongwon Suh Manager, Augmented Social Cognition Area echi@parc.com Social Search and Sensemaking: Wednesday, 4:30 PM - 6:00 PM Annotate Once, Appear Anywhere: Collective Foraging for Snippets of Interest Using Palo Alto Research Center Paragraph Fingerprinting Lichan Hong, Ed H. Chi With a Little Help from My Friends: Examining Research Center, Germany * Intern from Knowledge Media the Impact of Social Annotations in Sensemaking Tasks Les Nelson, Christoph Held, Peter Pirolli, Lichan Hong, Diane Schiano, Ed H. Chi

Social Search Survey [Brynn Evans, Ed H. Chi, CSCW2008] Help understand the   importance of: –  social cues and information exchanges –  vocabulary problems –  distribution and organization CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 2

TagSearch Exploratory Focus 3 kinds of search 59% 28% 13% informational navigational transactional You roughly know what you want You know what you want and where it is You know what you want to do but don’t know how to find it Difficult for existing search engines Existing search engines are OK Opportunity 3 CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 3

Research Motivation Social search systems:   –  Search and exploration services informed by human/group judgments and attention data. –  Social bookmarks and tags is a rich source of this data. Key Problems:   –  Coverage and participation –  Tag keyword ambiguity –  Spam and noise –  Chris Sherman, http://searchenginewatch.com/ showPage.html?page=3623153 CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 4

Using Information Theory to Model Social Tagging [Ed H. Chi, Todd Mytkowicz, Hypertext 2008] Topics  Concepts  Documents  Users  Noise  Tags  Decoding  Encoding  T1…Tn  5 CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 5

I(Doc; Tag) Tags contain less   information about documents and vice versa over time Source: del.icio.us (Chi & Mytkowicz, Hypertext2008) CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 6

Social Tagging Creates Noise •  Synonyms •  Misspellings •  Morphologies People use different tag words to express similar concepts. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 7

Use Semantic Analysis to Reduce Noise Semantic Similarity Graph Web Tools Reference Guide Howto Tutorial Tips Help Tutorials Tip Tricks CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 9

MapReduce Implementation Tags URLs P(URL|Tag) P(Tag|URL) Spreading Activation in a bigraph     Computation over a very large data set –  150 Million+ bookmarks CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 10

TagSearch Architecture  Database  Lucene  • Delicious  • P(URL|Tag)  • Serve up search  results  • Ma.gnolia  • P(Tag|URL)  • Tuples of  • Pre‐computed  • Well deﬁned APIs  • Other social cues  • Bayesian Network  bookmarks  paRerns in a fast  Inference  index  • [User, URL, Tags,  Time]  Crawling  MapReduce  Web Server  Web  Server  UI  Search  Frontend  Results  •  MapReduce: months of computaVon to a single day  •  Development of novel scoring funcVon   CHI2009 MrTaggy TagSEarch– © 2008 Palo Alto Research Center Inc. 11

Interlude: A Word on Exploratory Search User lack sufficient knowledge to define the   problem and search space -- ill-structured [Marchionini, 2006] Novices vs. experts   –  A problem may be ill-structured for a novice; –  But it’s well-structured for a seasoned expert. –  Implication: Experts might get less benefit from an exploratory search system. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 12

Experiment Design 2 interface x 3 task domain design   –  2 Interface (between-subjects) »  Exploratory vs. Baseline –  3 task domains (within-subjects) »  Future Architecture, Global Warming, Web Mashups 30 Subjects (22 male, 8 female)   –  Intermediate or advanced computer and web search skills –  Half assigned Exploratory, half Baseline. For each domain, single block with 3 task types:   –  Easy and Difficult Page Collection Task [6min each] –  Summarization Task [12min] –  Keyword Generation Task [2min] CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 15

Procedure [2 hours] Prior Knowledge Test     1st Task Domain –  With easy and difficult page collection tasks, summarization and keyword generation task. –  NASA cognitive load questionnaire 2nd Task Domain   –  Same battery of tasks and cognitive load questionaire 3rd Task Domain     Experimental Survey CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 18

Results: Interaction Behaviors Number of Queries   –  Effect of Interface on number of queries (p < .01) »  Exploratory (M=7.81) > Baseline (M=3.77) Time Taken   –  Effect of Interface on time taken (p < .01) »  Exploratory (7.7min) > Baseline (6.6min) CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 19

Results: Page Collection Task Measure of # of pages collected –  Effects of Task Domain (p<.01) and Task Difficulty (p<.05) –  Interaction effect of Interface by Task Domain (p<.05), with Exploratory interface performing better in the Web Mashup domain –  For relevance scores, similar patterns. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 20

Results: Summarization Tasks –  Quality of summarization scored (Cohen’s Kappa=0.7) –  ANCOVA with Prior Knowledge as covariate –  Exploratory Interface scored higher in Future Architecture (p<.05) and Global Warming (p<.05) –  For Web Mashup, Prior Knowledge correlated positively with performance (r=.51) CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 21

Results: Keyword Generation Tasks –  ANCOVA showed Exploratory > Baseline for Future Architecture (p<.05) and Web Mashups (p<.01), but not for Global Warming. –  Linear model between PK and # of keyword generated for Baseline showed mean slope = 0.32 and significant (p<. 05) CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 22

Discussion Exploratory interface users:   –  performed more queries, –  took more time, –  wrote better summaries (in 2/3 domains), –  generated more relevant keywords (in 2/3 domains), and –  had a higher cognitive load. Suggestive of deeper engagement and better   learning.   Some evidence of scaffolding for novices in the keyword generation and summarization tasks. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 24

Limitations Minimum control for domain expertise:   –  Lack depth in the implication for performance. Pre-defined task domains:   –  Lack ecological validity. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 25

Summary Harnessing user-generated tags to enrich content   for social search   Weaknesses of social tagging systems is Tag Noise and Inconsistency –  Difficult to leverage for search –  Use data mining techniques to normalize and reduce noise –  Apply normalized tag data in new search algorithm Study suggest deeper user engagement in   exploration and better learning with MrTaggy CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 26

Thanks! Try it now! http://mrtaggy.com http://spartag.us http://wikidashboard.parc.com Contact: Ed H. Chi, Ph.D. Manager, Augmented Social Cognition Area echi@parc.com Our Blog: http://asc-parc.blogspot.com CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 27

http://wordle.net Information Foraging: Tuesday, 9:00 AM - 10:30 AM An Elementary Social Information Foraging Model Peter Pirolli Remembrance of Things Tagged: How Tagging Effort Affects Tag Production and Human Memory Raluca Budiu, Peter Pirolli, Lichan Hong Signpost from the Masses: Learning Effects in an Exploratory Social Tag Search Browser Yvonne Kammerer, Rowan Nairn, Peter Pirolli, Ed H. Chi Studying Wikipedia: Wednesday, 11:30 AM - 1:00 PM So You Know You’re Getting the Best Possible Information: A Tool that Increases Wikipedia Credibility Peter Pirolli, Evelin Wollny, Bongwon Suh What's in Wikipedia? Mapping Topics and Conflict Using Socially Annotated Category Structure Aniket Kittur, Ed H. Chi, Bongwon Suh Social Search and Sensemaking: Wednesday, 4:30 PM - 6:00 PM Annotate Once, Appear Anywhere: Collective Foraging for Snippets of Interest Using Paragraph Fingerprinting Lichan Hong, Ed H. Chi With a Little Help from My Friends: Examining the Impact of Social Annotations in Sensemaking Tasks Les Nelson, Christoph Held, Peter Pirolli, Lichan Hong, Diane Schiano, Ed H. Chi

Cognition: the ability to remember, think, and reason; the faculty of    knowing.  Social Cognition: the ability of a group to remember, think, and    reason; the construction of knowledge structures by a group.  –  (not quite the same as in the branch of psychology that studies the  cognitive processes involved in social interaction, though included)  Augmented Social Cognition: Supported by systems, the    enhancement  of the ability of a group to remember, think, and  reason; the system‐supported construction of knowledge  structures by a group.   Citation: Ed H. Chi. The Social Web: Opportunities for Research.  IEEE Computer, Sept 2008  2008-11-07 Ed H. Chi ASC Overview 29

Social Transparency create trust and attribution: •  Increase participation via attribution Collective Intelligence •  Increase credibility and trust with community feedback •  Reduce wiki risks TagSearch: Mining social data for automatic data clustering and organization: •  Better organization via user- assigned tags Higher Productivity via •  Better UI for browsing sharing Generic benefits: interesting contents Collective Intelligence •  Greater trust •  Recommendation instead of •  Better decision-making just search Intelligence that emerges •  Useful sharing of info •  Auto-organization thru from the collaboration and search social data competition of many foraging individuals Foundation: SparTag.us: sharing of •  Understanding of human interesting contents: cognition and behavior •  A notebook that automatically •  Data mining of social data organizes your reading •  Social sharing of important and interesting tidbits •  Viral sharing of highlighted and tagged paragraphs 30

CHI2009 MrTaggy Tag-based Search Browser Intro and Evaluation

0 comments

Notes on slide 1

1 Favorite