Information Foraging: Tuesday, 9:00 AM - 10:30 AM
An Elementary Social Information Foraging Model
Peter Pirolli
Remembranc...
Social Search Survey
[Brynn Evans, Ed H. Chi, CSCW2008]


     Help understand the
 
     importance of:
     –  social c...
TagSearch Exploratory Focus

                                             3 kinds of search

                      59%    ...
Research Motivation

     Social search systems:
 
     –  Search and exploration services informed by human/group
      ...
Using Information Theory to Model Social Tagging
[Ed H. Chi, Todd Mytkowicz, Hypertext 2008]




                         ...
I(Doc; Tag)

                                                                   Tags contain less
                        ...
Social Tagging Creates Noise



                                                               •  Synonyms
               ...
CHI2009 MrTaggyit now: http://mrtaggy.com
           Try TagSearch– © 2008 Palo Alto Research Center Inc.   8
Use Semantic Analysis to Reduce Noise


          Semantic Similarity Graph
                                         Web
 ...
MapReduce Implementation

                       Tags                                   URLs



                          ...
TagSearch
Architecture


                              Database
                                                          ...
Interlude: A Word on Exploratory Search

     User lack sufficient knowledge to define the
 
     problem and search spac...
Baseline Interface




CHI2009 MrTaggyit now: http://mrtaggy.com
           Try TagSearch– © 2008 Palo Alto Research Cente...
Exploratory Interface




CHI2009 MrTaggyit now: http://mrtaggy.com
           Try TagSearch– © 2008 Palo Alto Research Ce...
Experiment Design
     2 interface x 3 task domain design
 
     –  2 Interface (between-subjects)
         »  Explorator...
Page Collection Tasks [6min each]




          CHI2009 MrTaggyit now: http://mrtaggy.com
                     Try TagSear...
Summarization Tasks [12min each]




         CHI2009 MrTaggyit now: http://mrtaggy.com
                    Try TagSearch–...
Procedure [2 hours]

   Prior Knowledge Test
 
  1st Task Domain
     –  With easy and difficult page collection tasks, ...
Results: Interaction Behaviors

     Number of Queries
 
     –  Effect of Interface on number of queries (p < .01)
     ...
Results: Page Collection Task
      Measure of # of pages collected




  –  Effects of Task Domain (p<.01) and Task Diffi...
Results: Summarization Tasks

                                                           –  Quality of summarization
     ...
Results: Keyword Generation Tasks

                                                                –  ANCOVA showed
      ...
Results: Cognitive Load




           –  Exploratory > Baseline (p<.05)



           CHI2009 MrTaggyit now: http://mrtag...
Discussion

     Exploratory interface users:
 
     –    performed more queries,
     –    took more time,
     –    wro...
Limitations

     Minimum control for domain expertise:
 
     –  Lack depth in the implication for performance.
     Pre...
Summary
   Harnessing user-generated tags to enrich content
 
   for social search
  Weaknesses of social tagging system...
Thanks!
  Try it now!
  http://mrtaggy.com
  http://spartag.us
  http://wikidashboard.parc.com

  Contact:
  Ed H. Chi, Ph...
http://wordle.net




Information Foraging: Tuesday, 9:00 AM - 10:30 AM
An Elementary Social Information Foraging Model
Pe...
Cognition:
the
ability
to
remember,
think,
and
reason;
the
faculty
of

 

     knowing.

     Social
Cognition:
the
abili...
Social Transparency create
                                                                                          trust...
Upcoming SlideShare
Loading in …5
×

CHI2009 MrTaggy Tag-based Search Browser Intro and Evaluation

1,812 views

Published on

CHI2009 research talk on MrTaggy: a Tag-based Search Browser, contains Introduction of the system and Evaluation of the interface

Published in: Technology, Design
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,812
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
21
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

CHI2009 MrTaggy Tag-based Search Browser Intro and Evaluation

  1. 1. Information Foraging: Tuesday, 9:00 AM - 10:30 AM An Elementary Social Information Foraging Model Peter Pirolli Remembrance of Things Tagged: How Tagging Effort Affects Tag Production and Human Signpost from the Masses: Memory Raluca Budiu, Peter Pirolli, Lichan Hong Signpost from the Masses: Learning Effects in an Exploratory Social Tag Search Browser Learning Effects in an Exploratory Yvonne Kammerer, Rowan Nairn, Peter Pirolli, Ed H. Chi StudyingSocial Tag Search Browser Wikipedia: Wednesday, 11:30 AM - 1:00 PM So You Know You’re Getting the Best Possible Information: A Tool that Increases Wikipedia Credibility Peter Pirolli, Evelin Wollny, Bongwon Rowan Nairn, Peter Pirolli, Ed H. Chi Yvonne Kammerer*, Suh What's in Wikipedia? Mapping Topics and Conflict Using Socially Annotated Category Contact: Structure Ed H. Chi, Ph.D. Aniket Kittur, Ed H. Chi, Bongwon Suh Manager, Augmented Social Cognition Area echi@parc.com Social Search and Sensemaking: Wednesday, 4:30 PM - 6:00 PM Annotate Once, Appear Anywhere: Collective Foraging for Snippets of Interest Using Palo Alto Research Center Paragraph Fingerprinting Lichan Hong, Ed H. Chi With a Little Help from My Friends: Examining Research Center, Germany * Intern from Knowledge Media the Impact of Social Annotations in Sensemaking Tasks Les Nelson, Christoph Held, Peter Pirolli, Lichan Hong, Diane Schiano, Ed H. Chi
  2. 2. Social Search Survey [Brynn Evans, Ed H. Chi, CSCW2008] Help understand the   importance of: –  social cues and information exchanges –  vocabulary problems –  distribution and organization CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 2
  3. 3. TagSearch Exploratory Focus 3 kinds of search 59% 28% 13% informational navigational transactional You roughly know what you want You know what you want and where it is You know what you want to do but don’t know how to find it Difficult for existing search engines Existing search engines are OK Opportunity 3 CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 3
  4. 4. Research Motivation Social search systems:   –  Search and exploration services informed by human/group judgments and attention data. –  Social bookmarks and tags is a rich source of this data. Key Problems:   –  Coverage and participation –  Tag keyword ambiguity –  Spam and noise –  Chris Sherman, http://searchenginewatch.com/ showPage.html?page=3623153 CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 4
  5. 5. Using Information Theory to Model Social Tagging [Ed H. Chi, Todd Mytkowicz, Hypertext 2008] Topics
 Concepts
 Documents
 Users
 Noise
 Tags
 Decoding
 Encoding
 T1…Tn
 5 CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 5
  6. 6. I(Doc; Tag) Tags contain less   information about documents and vice versa over time Source: del.icio.us (Chi & Mytkowicz, Hypertext2008) CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 6
  7. 7. Social Tagging Creates Noise •  Synonyms •  Misspellings •  Morphologies People use different tag words to express similar concepts. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 7
  8. 8. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 8
  9. 9. Use Semantic Analysis to Reduce Noise Semantic Similarity Graph Web Tools Reference Guide Howto Tutorial Tips Help Tutorials Tip Tricks CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 9
  10. 10. MapReduce Implementation Tags URLs P(URL|Tag) P(Tag|URL) Spreading Activation in a bigraph     Computation over a very large data set –  150 Million+ bookmarks CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 10
  11. 11. TagSearch
Architecture
 Database
 Lucene
 • Delicious
 • P(URL|Tag)
 • Serve
up
search
 results
 • Ma.gnolia
 • P(Tag|URL)
 • Tuples
of
 • Pre‐computed
 • Well
defined
APIs
 • Other
social
cues
 • Bayesian
Network
 bookmarks
 paRerns
in
a
fast
 Inference
 index
 • [User,
URL,
Tags,
 Time]
 Crawling
 MapReduce
 Web
Server
 Web
 Server
 UI
 Search
 Frontend
 Results
 •  MapReduce:
months
of
computaVon
to
a
single
day
 •  Development
of
novel
scoring
funcVon

 CHI2009 MrTaggy TagSEarch– © 2008 Palo Alto Research Center Inc. 11
  12. 12. Interlude: A Word on Exploratory Search User lack sufficient knowledge to define the   problem and search space -- ill-structured [Marchionini, 2006] Novices vs. experts   –  A problem may be ill-structured for a novice; –  But it’s well-structured for a seasoned expert. –  Implication: Experts might get less benefit from an exploratory search system. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 12
  13. 13. Baseline Interface CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 13
  14. 14. Exploratory Interface CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 14
  15. 15. Experiment Design 2 interface x 3 task domain design   –  2 Interface (between-subjects) »  Exploratory vs. Baseline –  3 task domains (within-subjects) »  Future Architecture, Global Warming, Web Mashups 30 Subjects (22 male, 8 female)   –  Intermediate or advanced computer and web search skills –  Half assigned Exploratory, half Baseline. For each domain, single block with 3 task types:   –  Easy and Difficult Page Collection Task [6min each] –  Summarization Task [12min] –  Keyword Generation Task [2min] CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 15
  16. 16. Page Collection Tasks [6min each] CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 16
  17. 17. Summarization Tasks [12min each] CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 17
  18. 18. Procedure [2 hours] Prior Knowledge Test     1st Task Domain –  With easy and difficult page collection tasks, summarization and keyword generation task. –  NASA cognitive load questionnaire 2nd Task Domain   –  Same battery of tasks and cognitive load questionaire 3rd Task Domain     Experimental Survey CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 18
  19. 19. Results: Interaction Behaviors Number of Queries   –  Effect of Interface on number of queries (p < .01) »  Exploratory (M=7.81) > Baseline (M=3.77) Time Taken   –  Effect of Interface on time taken (p < .01) »  Exploratory (7.7min) > Baseline (6.6min) CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 19
  20. 20. Results: Page Collection Task Measure of # of pages collected –  Effects of Task Domain (p<.01) and Task Difficulty (p<.05) –  Interaction effect of Interface by Task Domain (p<.05), with Exploratory interface performing better in the Web Mashup domain –  For relevance scores, similar patterns. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 20
  21. 21. Results: Summarization Tasks –  Quality of summarization scored (Cohen’s Kappa=0.7) –  ANCOVA with Prior Knowledge as covariate –  Exploratory Interface scored higher in Future Architecture (p<.05) and Global Warming (p<.05) –  For Web Mashup, Prior Knowledge correlated positively with performance (r=.51) CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 21
  22. 22. Results: Keyword Generation Tasks –  ANCOVA showed Exploratory > Baseline for Future Architecture (p<.05) and Web Mashups (p<.01), but not for Global Warming. –  Linear model between PK and # of keyword generated for Baseline showed mean slope = 0.32 and significant (p<. 05) CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 22
  23. 23. Results: Cognitive Load –  Exploratory > Baseline (p<.05) CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 23
  24. 24. Discussion Exploratory interface users:   –  performed more queries, –  took more time, –  wrote better summaries (in 2/3 domains), –  generated more relevant keywords (in 2/3 domains), and –  had a higher cognitive load. Suggestive of deeper engagement and better   learning.   Some evidence of scaffolding for novices in the keyword generation and summarization tasks. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 24
  25. 25. Limitations Minimum control for domain expertise:   –  Lack depth in the implication for performance. Pre-defined task domains:   –  Lack ecological validity. CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 25
  26. 26. Summary Harnessing user-generated tags to enrich content   for social search   Weaknesses of social tagging systems is Tag Noise and Inconsistency –  Difficult to leverage for search –  Use data mining techniques to normalize and reduce noise –  Apply normalized tag data in new search algorithm Study suggest deeper user engagement in   exploration and better learning with MrTaggy CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 26
  27. 27. Thanks! Try it now! http://mrtaggy.com http://spartag.us http://wikidashboard.parc.com Contact: Ed H. Chi, Ph.D. Manager, Augmented Social Cognition Area echi@parc.com Our Blog: http://asc-parc.blogspot.com CHI2009 MrTaggyit now: http://mrtaggy.com Try TagSearch– © 2008 Palo Alto Research Center Inc. 27
  28. 28. http://wordle.net Information Foraging: Tuesday, 9:00 AM - 10:30 AM An Elementary Social Information Foraging Model Peter Pirolli Remembrance of Things Tagged: How Tagging Effort Affects Tag Production and Human Memory Raluca Budiu, Peter Pirolli, Lichan Hong Signpost from the Masses: Learning Effects in an Exploratory Social Tag Search Browser Yvonne Kammerer, Rowan Nairn, Peter Pirolli, Ed H. Chi Studying Wikipedia: Wednesday, 11:30 AM - 1:00 PM So You Know You’re Getting the Best Possible Information: A Tool that Increases Wikipedia Credibility Peter Pirolli, Evelin Wollny, Bongwon Suh What's in Wikipedia? Mapping Topics and Conflict Using Socially Annotated Category Structure Aniket Kittur, Ed H. Chi, Bongwon Suh Social Search and Sensemaking: Wednesday, 4:30 PM - 6:00 PM Annotate Once, Appear Anywhere: Collective Foraging for Snippets of Interest Using Paragraph Fingerprinting Lichan Hong, Ed H. Chi With a Little Help from My Friends: Examining the Impact of Social Annotations in Sensemaking Tasks Les Nelson, Christoph Held, Peter Pirolli, Lichan Hong, Diane Schiano, Ed H. Chi
  29. 29. Cognition:
the
ability
to
remember,
think,
and
reason;
the
faculty
of
   knowing.
 Social
Cognition:
the
ability
of
a
group
to
remember,
think,
and
   reason;
the
construction
of
knowledge
structures
by
a
group.
 –  (not
quite
the
same
as
in
the
branch
of
psychology
that
studies
the
 cognitive
processes
involved
in
social
interaction,
though
included)
 Augmented
Social
Cognition:
Supported
by
systems,
the
   enhancement

of
the
ability
of
a
group
to
remember,
think,
and
 reason;
the
system‐supported
construction
of
knowledge
 structures
by
a
group.

 Citation:
Ed
H.
Chi.
The
Social
Web:
Opportunities
for
Research.

IEEE
Computer,
Sept
2008
 2008-11-07 Ed H. Chi ASC Overview 29
  30. 30. Social Transparency create trust and attribution: •  Increase participation via attribution Collective Intelligence •  Increase credibility and trust with community feedback •  Reduce wiki risks TagSearch: Mining social data for automatic data clustering and organization: •  Better organization via user- assigned tags Higher Productivity via •  Better UI for browsing sharing Generic benefits: interesting contents Collective Intelligence •  Greater trust •  Recommendation instead of •  Better decision-making just search Intelligence that emerges •  Useful sharing of info •  Auto-organization thru from the collaboration and search social data competition of many foraging individuals Foundation: SparTag.us: sharing of •  Understanding of human interesting contents: cognition and behavior •  A notebook that automatically •  Data mining of social data organizes your reading •  Social sharing of important and interesting tidbits •  Viral sharing of highlighted and tagged paragraphs 30

×