The CrowdSearch framework

418 views
371 views

Published on

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
418
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

The CrowdSearch framework

  1. 1. + A Framework for Crowdsourced Multimedia Processing and Querying Alessandro Bozzon, Ilio Catallo, Eleonora Ciceri, Piero Fraternali, Davide Martinenghi, Marco Tagliasacchi 0
  2. 2. + 1 CUbRIK Project  CUbRIK is a research project financed by the European Union  Goals:  Advance the architecture of multimedia search  Exploit the human contribution in multimedia search  Use open-source components provided by the community  Start up a search business ecosystem  http://www.cubrikproject.eu/
  3. 3. + 2 Humans in Multimedia Information Retrieval  Problem: the uncertainty of analysis algorithms leads to low confidence results and conflicting opinions on automatically extracted features  Solution: humans have superior capacity for understanding the content of audiovisual material  State of the art: humans replace automatic feature extraction processes (human annotations)  Our contribution: integration of human judgment and algorithms  Goal: improve the performance of multimedia content processing
  4. 4. + Example of CUbRIK Human-enhanced 3 computation: Trademark Logo Detection  Problem statement: identifying occurrences of trademark logos in a video collection through keyword-based queries  Special case of the classic problem of object recognition  Use case: a professional user wants to retrieve all the occurrences of logos in a large collection of video clips  Applications: rating effectiveness of advertising, subliminal advertising detection, automatic annotation, trademark violation detection
  5. 5. + 4 Trademark Logo Detection: problems in automatic logo detection  Problems in automatic logo detection:  Object recognition is affected by the quality of the input set of images  Uncertain matches, i.e., the ones with low matching score, could not contain the searched logo
  6. 6. + 5 Trademark Logo Detection: contribution of human computation  Contribution in human computation  Filter the input logos, eliminating the irrelevant ones  Segment the input logos  Validate the matching results
  7. 7. + 6 Trademark Logo Detection: pipeline
  8. 8. + 7 The CrowdSearch framework for HC task management
  9. 9. + 8 CrowdSearch framework in the Logo detection application Problem solving process Process Task Crowd Task Types of tasks • Automatic tasks • Crowd tasks: tasks that are executed by an open-ended community of performers Crowd Task
  10. 10. + 9 Community of Performers Content edges, e.g., IS-A, part.of Content elements The application is deployed as a Facebook application Seed community Information Technology Performer to content department of Politecnico di edges, e.g., topical group membership Milano Performers edges, e.g., friendship, Task propagation weak ties Performers Each user in the seed community can propagate tasks through the social networks
  11. 11. + 10 Design of “Validate Logo Images” The “LIKE” task variant requires to choose relevant logos among a set of not filtered images Human Task Design The “ADD”task variant requires to add new relevant image URLs Please add new relevant logos URL… Send
  12. 12. + 11 People to task matching & Task AssignmentTask Deployment Criteria Execution criteria Constraints of task execution Content Affinity Criteria Time budget for the experiment Execution Criteria Content Affinity criteria Query on a representation of the users’ capacities • Current state: manual selection of users People to • Future work: Geocultural affinity task matching Questions are dispatched to the crowd according to the user experience in answering questions • Expert user: an user that has already answered to three questions Task New users answer to “LIKE” questions assignment Expert users answer to “LIKE”+“ADD” questions
  13. 13. + 12 Task execution Task execution “LIKE” task variant “ADD” task variant
  14. 14. + 13 Output aggregation “LIKE” task variants Top-5 rated logos are selected as relevant logos Task “ADD” task variants execution New images are fed back to the LIKE tasks Task outputs Task output Output aggregation
  15. 15. + 14 Experimental evaluation  Three experimental settings:  No human intervention  Logo validation performed by two domain experts  Inclusion of the actual crowd knowledge  Crowd involvement  40 people involved  50 task instances generated  70 collected answers
  16. 16. + 15 Experimental evaluation 1 0.9 0.8 Crowd 0.7 Experts 0.6 Experts Recall Experts 0.5 Aleve 0.4 Crowd Chunky 0.3 No Crowd Shout 0.2 Crowd No Crowd 0.1 0 No Crowd 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision
  17. 17. + 16 Experimental evaluation 1 0.9 0.8 Precision decreases Crowd 0.7 Experts 0.6 Reasons for the wrong inclusion Experts Recall Experts • Geographical location of the users 0.5 Aleve • Expertise of the involved users 0.4 Crowd Chunky 0.3 No Crowd Shout 0.2 Crowd No Crowd 0.1 0 No Crowd 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision
  18. 18. + 17 Experimental evaluation 1 Precision decreases • Similarity between two 0.9 logos in the data set 0.8 Crowd 0.7 Experts 0.6 Experts Recall Experts 0.5 Aleve 0.4 Crowd Chunky 0.3 No Crowd Shout 0.2 Crowd No Crowd 0.1 0 No Crowd 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision
  19. 19. + 18 Future directions  Task design:  Implement new task types (tag / comment / like / add / modify…)  Partition large task instances into several smaller instances dispatched to multiple users  Task assignment: study how to associate the most suitable request with the most appropriate user  Implement a ranking function on worker pool, based on the expertise, geocultural information and past work history of the performers  Task execution: multiple heterogeneous platforms (Facebook, LinkedIn, Twitter, stand-alone application)  More use cases:  Breaking news  Fashion trend

×