0
23. April 2013 1Introduction to Collaborative Web .Sergej Zerrzerr@L3S.de
Sergej Zerr 2Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration•Using Collabor...
Sergej Zerr 3CollaborationOften we need more than one handSometimes more than one brain“Why the Many Are Smarter Than the ...
Sergej Zerr 4Direct Collaboration in WWW can be used:Collaborative tagging,Favorite assignments,Click logs,Data gathering,...
Sergej Zerr 5Social ComputerExpertUtilityMasses# of contributors10 10,000+Equivalent, or greater, utilityunder the curve
Sergej Zerr 6Social ComputerExpert $$$$UtilityMasses $# of contributors10 10,000+4,000 experts80,000 articles200 years to ...
Sergej Zerr 7Collaboration: Paradoxhttp://spectrum.ieee.org/computing/hardware/multicore-is-bad-news-for-supercomputersCol...
Sergej Zerr 8CollaborationCriteria DescriptionDiversity of opinion Each person should have private information.Independenc...
Sergej Zerr 9Groupthink Symptoms:Irving Lester Janis (26 May 1918 - 15 November 1990)• Illusion of invulnerability• Collec...
Sergej Zerr 10Collaboration“The best collective decisions are the product ofdisagreement and contest, not consensus or com...
Sergej Zerr 11Many Complementary Games• ESP Game: label images Image retrieval by text• Squigl: match the labels to areas...
Sergej Zerr 12Re-Using Human Power
Sergej Zerr 13Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration•Using Collabo...
Sergej Zerr 14Social ComputerHuman can (yet) solve some tasksmore efficient and/or accurateas a machine would do.• Captcha...
Sergej Zerr 15Social Computer – Good QuestionMatthew Lease and Omar Alonso: http://de.slideshare.net/mattlease/crowdsourci...
Sergej Zerr 16Social Computer – Good InterfaceMatthew Lease and Omar Alonso: http://de.slideshare.net/mattlease/crowdsourc...
Sergej Zerr 17L3S Research – “What about image privacy?”Sergej Zerr , Stefan Siersdorfer , Jonathon Hare , Elena Demidova ...
Sergej Zerr 18L3S ResearchGathering average community notion of privacy• We crawled “most recently uploaded” Flickr photos...
Sergej Zerr 19Sampling Data from Social Web: Tags0500001000001500002000002500001 10 100 1000#photos#Tags per PhotoTags Dis...
Sergej Zerr 20Sampling Data from Social Web: User activitySimple random sample can result in a set dominated by few power ...
Sergej Zerr 21Inter–Rater Reliability: What about ambiguity?Nederland, netherlands, holland, dutchRotterdam, wielrennen, c...
Sergej Zerr 22Inter–Rater Reliability “Where is the cat?”X XV V V VXX XVV VX X X X X XV VX XX X Results
Sergej Zerr 23Inter–Rater Reliability “Where is the cat?”X XXX XV V V VVV VX X X X X XV VX XX X Results• Ideas• Introduce ...
Attractive?1 10 10 01 01 01 1Sergej Zerr 24Inter–Rater Reliability:AtotalYes NoB Yes 2 1 3No 2 1 3total 4 2 6Naive approac...
Sergej Zerr 25Inter–Rater Reliability: Cohen‟s Kappa(1960)Idea: We need to remove agreement achived just by chanceeeappp...
Sergej Zerr 26Inter–Rater Reliability: Missing ValuesIdea: Use partial ratings to estimate marginal probability onlyAtotal...
Sergej Zerr 27Inter–Rater Reliability: Extentions• Multiple Raters/Categories:• Fleiss 1971 – Average over random pairs of...
Sergej Zerr 28Inter–Rater Reliability: Kappa InterpretationsKochKappa Strenght of Agreement<0.0 Poor0.0 – 0.20 Slight0.21 ...
Sergej Zerr 29Useful human power for annotating the Web• 5000 people playing simultaneously can label all images on Google...
Sergej Zerr 30Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration• Using Collab...
Sergej Zerr 31What is relevant?
Sergej Zerr 32Search and Sensemaking ProcessQueryResult listCan collaboration improve the sensemaking process at any step?...
Sergej Zerr 33What are the typical collaborative search tasks?0.005.0010.0015.0020.0025.0030.00TravelplanningGeneralshoppi...
Sergej Zerr 34Collaborative Search:• Query formulation• Search process/browsing• Save/Bookmark• Annotate/OrganizeHow to su...
Sergej Zerr 35Indirect Collaboration on Web:Google uses search/click logs. For PageRank algorithm each link to a pageserve...
Sergej Zerr 36Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration• Using Collab...
Sergej Zerr 37Software support for Co-located search (CoSearch).Amershi, S., Morris, M. CoSearch: System for colocated col...
Sergej Zerr 38Software support for Co-located search (CoSearch).Amershi, S., Morris, M. CoSearch: System for colocated col...
Sergej Zerr 39Spatial distributed Search (SearchTogether)Morris, M.R. & Horvitz, E. SearchTogether: An Interface for Colla...
Sergej Zerr 40Spatial distributed Search (SearchTogether)Morris, M.R. & Horvitz, E. SearchTogether: An Interface for Colla...
Sergej Zerr 41Improvement Through Collaborative RankingAgrahri, A., Manickam, D., Riedl, J. Can people collaborate to impr...
Sergej Zerr 42WeSearch: Collaborative Sensemaking (Hardware support)Meredith Ringel Morris, Jarrod Lombardo, and Daniel Wi...
Sergej Zerr 43WeSearch: Collaborative SensemakingMeredith Ringel Morris, Jarrod Lombardo, and Daniel Wigdor. 2010. WeSearc...
Sergej Zerr 44Hardware support for Co-located(TeamSearch).Morris, M.R., Paepcke, A., and Winograd, T. Team-Search: Compari...
Sergej Zerr 45Hardware support for Co-located(TeamSearch).Morris, M.R., Paepcke, A., and Winograd, T. Team-Search: Compari...
Sergej Zerr 46Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration• Using Collab...
23. Nov. 2010Sergej Zerr
Sergej Zerr 48Smalltalk, what should be the properties of a perfectcollaborative system?
Upcoming SlideShare
Loading in...5
×

Collaborative search intro 2013

98

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
98
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Collaborative search intro 2013"

  1. 1. 23. April 2013 1Introduction to Collaborative Web .Sergej Zerrzerr@L3S.de
  2. 2. Sergej Zerr 2Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration•Using Collaborative Data• Gathering Data from Social Web / Mechanical Turk• Inter Rater Agreement• Search and Sensemaking processes, the overview• Collaboration opportunities in (Web) search• Overview Papers• Collaborative Search, Search result relevance judgments,..•Small experiment• Can we collaborate?• Discussion
  3. 3. Sergej Zerr 3CollaborationOften we need more than one handSometimes more than one brain“Why the Many Are Smarter Than the Few andHow Collective Wisdom Shapes Business,Economies, Societies and Nations”James Suroewicki
  4. 4. Sergej Zerr 4Direct Collaboration in WWW can be used:Collaborative tagging,Favorite assignments,Click logs,Data gathering,Reccomendations,ect., ect.,ect….Tags: Rainbow, Sea, Island, Green, Palmtree, Maui
  5. 5. Sergej Zerr 5Social ComputerExpertUtilityMasses# of contributors10 10,000+Equivalent, or greater, utilityunder the curve
  6. 6. Sergej Zerr 6Social ComputerExpert $$$$UtilityMasses $# of contributors10 10,000+4,000 experts80,000 articles200 years to developAnnual Updates100,000 amateurs1.6 Million articles5 years to developReal-Time Updates
  7. 7. Sergej Zerr 7Collaboration: Paradoxhttp://spectrum.ieee.org/computing/hardware/multicore-is-bad-news-for-supercomputersCollaborative work needs to be managedefficientlyKasparov won against the world
  8. 8. Sergej Zerr 8CollaborationCriteria DescriptionDiversity of opinion Each person should have private information.Independence Peoples opinions arent determined by the opinions ofthose around them.Decentralization People are able to specialize and draw on localknowledge.Aggregation Some mechanism exists for turning private judgmentsinto a collective
  9. 9. Sergej Zerr 9Groupthink Symptoms:Irving Lester Janis (26 May 1918 - 15 November 1990)• Illusion of invulnerability• Collective rationalization• Belief in inherent morality• Stereotyped views of out-groupshttp://en.wikipedia.org/wiki/Groupthink• Direct pressure on dissenters• Self-censorship• Illusion of unanimity• Self-appointed „mindguards‟
  10. 10. Sergej Zerr 10Collaboration“The best collective decisions are the product ofdisagreement and contest, not consensus or compromise.”“The best way for a group to be smart is for each person in itto think and act as independently as possible.”
  11. 11. Sergej Zerr 11Many Complementary Games• ESP Game: label images Image retrieval by text• Squigl: match the labels to areas Object recognition• Matchin: find the better image Image ranking• FlipIt: memory with similar imagesNear duplicate detection• Other areas covered as well: label songs, find synonyms, describe videos• See: www.gwap.com by Luis von Ahn
  12. 12. Sergej Zerr 12Re-Using Human Power
  13. 13. Sergej Zerr 13Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration•Using Collaborative Data• Gathering Data from Social Web / Mechanical Turk• Inter Rater Agreement• Search and Sensemaking processes, the overview• Collaboration opportunities in (Web) search• Overview Papers• Collaborative Search, Search result relevance judgments,..•Small experiment• Can we collaborate?• DiscussionTwo different images that share the same ESP labels:man and woman
  14. 14. Sergej Zerr 14Social ComputerHuman can (yet) solve some tasksmore efficient and/or accurateas a machine would do.• Captcha• Classification (OCR)• Image tagging• Speech recognition
  15. 15. Sergej Zerr 15Social Computer – Good QuestionMatthew Lease and Omar Alonso: http://de.slideshare.net/mattlease/crowdsourcing-for-search-evaluation-and-socialalgorithmic-search„Mr. Burns saw Homer with the binoculars“
  16. 16. Sergej Zerr 16Social Computer – Good InterfaceMatthew Lease and Omar Alonso: http://de.slideshare.net/mattlease/crowdsourcing-for-search-evaluation-and-socialalgorithmic-searchThrow the coin and tell us the result• Head?• Tail?Results• Head 61• Tail 39People often tendjust to select thefirst option Better: Some preliminary textual answer• Coin type?• Head or tail.
  17. 17. Sergej Zerr 17L3S Research – “What about image privacy?”Sergej Zerr , Stefan Siersdorfer , Jonathon Hare , Elena Demidova Privacy-Aware Image Classification and Search , SIGIR„12
  18. 18. Sergej Zerr 18L3S ResearchGathering average community notion of privacy• We crawled “most recently uploaded” Flickr photos (2 Months)• Started a social annotation game (over the course of 2 weeks)• 81 users (colleagues, social networks , forum users) , 6 teamsSergej Zerr , Stefan Siersdorfer , Jonathon Hare , Elena Demidova Privacy-Aware Image Classification and Search , SIGIR„12
  19. 19. Sergej Zerr 19Sampling Data from Social Web: Tags0500001000001500002000002500001 10 100 1000#photos#Tags per PhotoTags Distribution of 2MioFlickr "Streetart" Images>75 tags per photo1-20 tags per photo
  20. 20. Sergej Zerr 20Sampling Data from Social Web: User activitySimple random sample can result in a set dominated by few power user
  21. 21. Sergej Zerr 21Inter–Rater Reliability: What about ambiguity?Nederland, netherlands, holland, dutchRotterdam, wielrennen, cycling, duckle grand depart, tour de france,Reklame, caravan, Funny Fotos
  22. 22. Sergej Zerr 22Inter–Rater Reliability “Where is the cat?”X XV V V VXX XVV VX X X X X XV VX XX X Results
  23. 23. Sergej Zerr 23Inter–Rater Reliability “Where is the cat?”X XXX XV V V VVV VX X X X X XV VX XX X Results• Ideas• Introduce „Honeypot“ – a set of ground truth objects and select only good raters• Select only raters who rate close to average rating.• Other startegies?
  24. 24. Attractive?1 10 10 01 01 01 1Sergej Zerr 24Inter–Rater Reliability:AtotalYes NoB Yes 2 1 3No 2 1 3total 4 2 6Naive approach: 3 cases out of 6 = 0.5 agreementKilem L. Gwet, Handbook of inter-rater reliability 2010A B
  25. 25. Sergej Zerr 25Inter–Rater Reliability: Cohen‟s Kappa(1960)Idea: We need to remove agreement achived just by chanceeeappp1ˆ49.010060*1004510040*10055ep75.01004035222112112211nnnnnnpaAtotalYes NoB Yes 35 20 55No 5 40 45total 40 60 10051.49.149.75.0ˆ Kilem L. Gwet, Handbook of inter-rater reliability 2010AtotalYes NoB Yes n11 n12No n21 n22total
  26. 26. Sergej Zerr 26Inter–Rater Reliability: Missing ValuesIdea: Use partial ratings to estimate marginal probability onlyAtotalYes No XB Yes 30 18 2 50No 5 34 3 42X 5 3 0 8total 40 55 5 100eeappp1ˆ431.010055*1004210040*10050ep74.)85(1003430)( 21212211xxxxannnnnnnp54.431.1431.74.0ˆ Kilem L. Gwet, Handbook of inter-rater reliability 2010AtotalYes No XB Yes n11 n12 nx1No n21 n22 nx2X n1x n2x 0total
  27. 27. Sergej Zerr 27Inter–Rater Reliability: Extentions• Multiple Raters/Categories:• Fleiss 1971 – Average over random pairs of raters for random objects• Adjustment for Ordinal and Interval Data, Weighting:• weight judgements using distances between categories.• Reduce Kappa Paradox:• Split judgements into more certain / marginal.• Measures: AC1, AC2(ordinal and interval data)• Check for statistical significance:• The number of categories and/or raters matters.Kilem L. Gwet, Handbook of inter-rater reliability 2010
  28. 28. Sergej Zerr 28Inter–Rater Reliability: Kappa InterpretationsKochKappa Strenght of Agreement<0.0 Poor0.0 – 0.20 Slight0.21 - 0.40 Fair0.41 - 0.60 Moderate0.61 - 0.80 Substantual0.81 - 100 Almost PerfectFleissKappa Strenght of Agreement0.0-0.40 Poor0.41 – 0.75 Intermediate to Good>0.75 ExcellentAltmanKappa Strenght of Agreement<0.20 Poor0.21 - 0.40 Fair0.41 - 0.60 Moderate0.61 - 0.80 Good0.81 - 100 Very GoodPlease note: These interpretations were proven to be usefull mostly in medical domain (diagnosis)Kilem L. Gwet, Handbook of inter-rater reliability 2010
  29. 29. Sergej Zerr 29Useful human power for annotating the Web• 5000 people playing simultaneously can label all images on Googlein 30 days!• Individual games in Yahoo! and MSN average over 5,000 players ata timePossible contributions: attach labels to images in other languages,categorize web pages into topics
  30. 30. Sergej Zerr 30Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration• Using Collaborative Data• Gathering Data from Social Web / Mechanical Turk• Inter Rater Agreement• Search and Sensemaking processes, the overview• Collaboration opportunities in (Web) search• Overview Papers• Collaborative Search, Search result relevance judgments,..•Small experiment• Can we collaborate?• Discussion
  31. 31. Sergej Zerr 31What is relevant?
  32. 32. Sergej Zerr 32Search and Sensemaking ProcessQueryResult listCan collaboration improve the sensemaking process at any step?Do you use collaborative search?Annotation/Organization
  33. 33. Sergej Zerr 33What are the typical collaborative search tasks?0.005.0010.0015.0020.0025.0030.00TravelplanningGeneralshoppingtasksLiteraturesearchTechnicalinformationFact finding SocialplanningMedicalinformationReal estateMorris, M.R. A Survey of Collaborative Web Search Practices. In Proceedings of 26th CHI Conference 2008Around 90% of Microsoft employees are engaged in collaborative searchactivities.• Watched over someone‟s shoulder ashe/she searched the Web, and suggestedalternate query terms.• E-mailed someone links to share the resultsof a Web search.• E-mailed someone a textual summary toshare the results of a Web search.• Called someone on the phone to tell themabout the results of a Web search.Morris, M.R. A Survey of Collaborative Web Search Practices. In Proceedings of 26th CHI Conference 2008
  34. 34. Sergej Zerr 34Collaborative Search:• Query formulation• Search process/browsing• Save/Bookmark• Annotate/OrganizeHow to support users in collaborative searching?- Ideas- Tools(Web 2.0)
  35. 35. Sergej Zerr 35Indirect Collaboration on Web:Google uses search/click logs. For PageRank algorithm each link to a pageserves as a vote for that page.Amazon uses search/click logs. For item recommendation similar users are theindirect voters for the product.ect., ect.
  36. 36. Sergej Zerr 36Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration• Using Collaborative Data• Gathering Data from Social Web / Mechanical Turk• Inter Rater Agreement• Search and Sensemaking processes, the overview• Collaboration opportunities in (Web) search• Overview Papers• Collaborative Search, Search result relevance judgments,..•Small experiment• Can we collaborate?• Discussion
  37. 37. Sergej Zerr 37Software support for Co-located search (CoSearch).Amershi, S., Morris, M. CoSearch: System for colocated collaborative Web search. In Proceedings of 26th CHI Conference 2008
  38. 38. Sergej Zerr 38Software support for Co-located search (CoSearch).Amershi, S., Morris, M. CoSearch: System for colocated collaborative Web search. In Proceedings of 26th CHI Conference 2008Qualitative criteria 5 point Likert scale (5=strongly agree)Was CoSearch easy to use? 3Were the colors useful? 4.5Was the query queue useful? 4Were color tabs useful? 4Mobile view useful? 4I would use CoSearch at work and in the school 4I would use CoSearch at home 3.5
  39. 39. Sergej Zerr 39Spatial distributed Search (SearchTogether)Morris, M.R. & Horvitz, E. SearchTogether: An Interface for Collaborative Web Search. In Proceedings of the UIST 2007• (a) integrating messaging• (b) query awareness,• (c) current results• (d) recommendationqueue• (e)(f)(g) search buttons• (h) page-specificmetadata• (i) toolbar• (j) browser
  40. 40. Sergej Zerr 40Spatial distributed Search (SearchTogether)Morris, M.R. & Horvitz, E. SearchTogether: An Interface for Collaborative Web Search. In Proceedings of the UIST 2007Qualitative criteria (5 points Likert scale) Collective tokensSearchTogether helped to complete the joint task 3.9SearchTogether is more effective than other tools 4.1• 38% af all result lists were theconsequence of using history• 70 positive and 9 negative ratings• 22 of 36 recommendation wereviewed by the recipients
  41. 41. Sergej Zerr 41Improvement Through Collaborative RankingAgrahri, A., Manickam, D., Riedl, J. Can people collaborate to improve the relevance of search results? In Proceedings of the ACM International Conference on Recommendation Systems 2008
  42. 42. Sergej Zerr 42WeSearch: Collaborative Sensemaking (Hardware support)Meredith Ringel Morris, Jarrod Lombardo, and Daniel Wigdor. 2010. WeSearch: supporting collaborative search and sensemaking on a tabletop display. In Proceedings of the 2010 ACM conference onComputer supported cooperative work (CSCW 10). ACM, New York, NY, USA, 401-410.
  43. 43. Sergej Zerr 43WeSearch: Collaborative SensemakingMeredith Ringel Morris, Jarrod Lombardo, and Daniel Wigdor. 2010. WeSearch: supporting collaborative search and sensemaking on a tabletop display. In Proceedings of the 2010 ACM conference onComputer supported cooperative work (CSCW 10). ACM, New York, NY, USA, 401-410.Qualitative criteria (7 point Likert scale)The system was easy to use 6High awareness af others„ activity 5Export record for later view was useful 6Prefer clip cutting instead of clip search 6
  44. 44. Sergej Zerr 44Hardware support for Co-located(TeamSearch).Morris, M.R., Paepcke, A., and Winograd, T. Team-Search: Comparing Techniques for Co-Present Collaborative Search of Digital Media.• Circles are categories:people, location, year, event
  45. 45. Sergej Zerr 45Hardware support for Co-located(TeamSearch).Morris, M.R., Paepcke, A., and Winograd, T. Team-Search: Comparing Techniques for Co-Present Collaborative Search of Digital Media.Qualitative criteria (7 points Likert scale) Collective tokens Parallel tokensI worked closely with the other members of mygroup to accomplish this task.5.75 4.88Members of the group communicated with eachother effectively.5.75 5The group worked effectively as a team on thistask.5.75 4.81
  46. 46. Sergej Zerr 46Outline• Collaborative Advantages• Wisdom of crowds• Conditions for a successful collaboration• Using Collaborative Data• Gathering Data from Social Web / Mechanical Turk• Inter Rater Agreement• Search and Sensemaking processes, the overview• Collaboration opportunities in (Web) search• Overview Papers• Collaborative Search, Search result relevance judgments,..•Small experiment• Can we collaborate?• Discussion
  47. 47. 23. Nov. 2010Sergej Zerr
  48. 48. Sergej Zerr 48Smalltalk, what should be the properties of a perfectcollaborative system?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×