CUbRIK at SoHuman 2012: A draw and-guess game to segment images


Published on

CUbRIK technology presented at SoHuman 2012 (Social Media For Human Computation)

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Searching and Manipulating multimedia contentLow confidence results, Conflicting opinion on features extracted automaticallySuperior capacity to understand complex contentCommon sense elicitation and content tagging
  • The need formultimedia content analysisSupport for high level queriesIdentification of trends in collections of images.
  • The proposed solutionHuman computation approachWith the High level abstraction capabilities of humans when working with visual content
  • CUbRIK at SoHuman 2012: A draw and-guess game to segment images

    1. 1. A Draw-and-Guess Game toSegment ImagesLuca Galli, Piero FraternaliDavide Martinenghi, Marco TagliasacchiPolitecnico di Milano, ItalyEmail: first.last@polimi.itJasminko NovakEuropean Institute for Participatory Media,GermanyEmail:
    2. 2. The CUbRIK Project● CUbRIK is a research project financed by the European Union● Goals: ● Advance the architecture of multimedia search ● Exploit the human contribution in multimedia search ● Use open-source components provided by the community ● Start up a search business ecosystem● Luca Galli
    3. 3. Humans in Multimedia Information Retrieval● Problem: the uncertainty of analysis algorithms leads to low confidence results and conflicting opinions on automatically extracted features● Solution: humans have superior capacity for understanding the content of audiovisual material ● State of the art: humans replace automatic feature extraction processes (human annotations) ● Our contribution: integration of human judgment and algorithms – Goal: improve the performance of multimedia content processing Luca Galli
    4. 4. 4 The Fashion Trend Mining Scenario● Problem statement: segment fashion images for mining trends based on visual features of garments (e.g. color and texture) Color descriptors coarse (sub-)image similarity Texture descriptors● Use case: identifying trends in collections of images of people and garments● Applications: retrieving similar garments, inspect clothing trends in image collections, analyzing trends change in the years Luca Galli
    5. 5. 5Issues of traditional approaches Problems in automatic feature detection:  Body part and feature recognition is affected by the posture of the portrayed subject ?  Problems arise when part of the body is occluded or in pictures with complex human poses Luca Galli
    6. 6. 6 Solution: Human Contribution Issues are solved by augmenting the capabilities of algorithms with human intervention.  Humans have superior capacity to understand complex content and to perform high level abstractions on visual representations  For these reasons, they are suitable for tasks such as: ● Body part recognition ● Garments segmentation ● Gender identification Luca Galli
    7. 7. The CUbRIK Architecture Content and users registration in the system Managing tasks and processing contents Provides front end for issuing queries and viewing results Search Engines used to access the content and annotations Luca Galli
    8. 8. 8Users Motivation: Games with a Purpose ● Users need motivation to complete a task in a human computation platform, such as monetary rewards, prestige, entertainment. ● Games with a Purpose (GWAPs) are digital games that generate useful data as a by-product of play. ● The design of a GWAP requires to create a game so that its structure encourages computation, correctness of the output and players retention. ● Introduced by Luis Von Ahn with the ESP Game, several GWAPs have been developed with a number of different purposes Luca Galli
    9. 9. 9The Fashion Trend Mining Pipeline Male, 24 Female, 22 Female?, ?? Luca Galli
    10. 10. 10 Gwap Design: Sketchness!● Multiplayer Puzzle Game● Established genre: Guess and Draw (Pictionary, iSketch…)● Inversion Problem Mechanic● Players take turns into drawing the shapes of objects inside an image in order to make the other players guess the underlying object.● Two different roles: Sketcher Guesser● Objective: Garments tagging, body part segmentation, garment segmentation Luca Galli
    11. 11. 11 Player Role: Sketcher Tag of the target object● The only player to see the low confidence image● “May” be asked to provide a tag for the image● Is asked to draw the contour of the object for which the tag is provided within the allotted time● Goal of the Sketcher is to let the other players guess the tag without providing any other hints than the contour Fashion image provided Contour of the garment by the pipeline provided by the player Luca Galli
    12. 12. 12 Player Role: Guesser● Any other player in the game● His/Her goal is to guess the object for which the Sketcher has provided the contour● Not allowed to draw on the whiteboard, just to type in the chat box the probable answer as fast as possible● Scoring: ● Sketcher: 10 pts + 1 for each guesser ● Guesser: 10 pts to the first, then Contour of the garment decreasing down to five provided by the Sketcher Luca Galli
    13. 13. 13 Task Injection: new approach to GWAPs● Traditional gwaps developed on ad-hoc basis and tailored for a specific task● Resulting gwaps may be not fun, unappealing to players or provide an experience that may still perceived by the users as work● The proposed GWAP exploits suitable modified game mechanics of a well known and appreciated category of games, called “drawing and guessing games” to implicitly solve segmentation problems while playing Luca Galli
    14. 14. 14 Future Directions ● The proposed game concept is still under development and there are still several open questions and future improvements:● Task Injection: ● Does the task injection approach guarantee the same level of entertainment and participation for the players as the original instance of the game? ● How to measure entertainment anyhow?● Output Aggregation: ● How the results provided by different players on the same input be processed and grouped together in order to obtain a unique result? ● Majority voting not suitable for lines!● Integration within CUbRIK’s Gaming Framework: ● Exploiting the features provided by CUbRIK in order to manage players and their abilities and provide retention and reward features through gamification techniques. Luca Galli