A draw and-guess game to segment images
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


A draw and-guess game to segment images






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds


Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Searching and Manipulating multimedia contentLow confidence results, Conflicting opinion on features extracted automaticallySuperior capacity to understand complex contentCommon sense elicitation and content tagging
  • The need formultimedia content analysisSupport for high level queriesIdentification of trends in collections of images.
  • The proposed solutionHuman computation approachWith the High level abstraction capabilities of humans when working with visual content

A draw and-guess game to segment images Presentation Transcript

  • 1. A Draw-and-Guess Game toSegment ImagesLuca Galli, Piero FraternaliDavide Martinenghi, Marco TagliasacchiPolitecnico di Milano, ItalyEmail: first.last@polimi.itJasminko NovakEuropean Institute for Participatory Media,GermanyEmail: j.novak@eipcm.org
  • 2. The CUbRIK Project● CUbRIK is a research project financed by the European Union● Goals: ● Advance the architecture of multimedia search ● Exploit the human contribution in multimedia search ● Use open-source components provided by the community ● Start up a search business ecosystem● http://www.cubrikproject.eu/ Luca Galli
  • 3. Humans in Multimedia Information Retrieval● Problem: the uncertainty of analysis algorithms leads to low confidence results and conflicting opinions on automatically extracted features● Solution: humans have superior capacity for understanding the content of audiovisual material ● State of the art: humans replace automatic feature extraction processes (human annotations) ● Our contribution: integration of human judgment and algorithms – Goal: improve the performance of multimedia content processing Luca Galli
  • 4. 4 The Fashion Trend Mining Scenario● Problem statement: segment fashion images for mining trends based on visual features of garments (e.g. color and texture) Color descriptors coarse (sub-)image similarity Texture descriptors● Use case: identifying trends in collections of images of people and garments● Applications: retrieving similar garments, inspect clothing trends in image collections, analyzing trends change in the years Luca Galli
  • 5. 5Issues of traditional approaches Problems in automatic feature detection:  Body part and feature recognition is affected by the posture of the portrayed subject ?  Problems arise when part of the body is occluded or in pictures with complex human poses Luca Galli
  • 6. 6 Solution: Human Contribution Issues are solved by augmenting the capabilities of algorithms with human intervention.  Humans have superior capacity to understand complex content and to perform high level abstractions on visual representations  For these reasons, they are suitable for tasks such as: ● Body part recognition ● Garments segmentation ● Gender identification Luca Galli
  • 7. The CUbRIK Architecture Content and users registration in the system Managing tasks and processing contents Provides front end for issuing queries and viewing results Search Engines used to access the content and annotations Luca Galli
  • 8. 8Users Motivation: Games with a Purpose ● Users need motivation to complete a task in a human computation platform, such as monetary rewards, prestige, entertainment. ● Games with a Purpose (GWAPs) are digital games that generate useful data as a by-product of play. ● The design of a GWAP requires to create a game so that its structure encourages computation, correctness of the output and players retention. ● Introduced by Luis Von Ahn with the ESP Game, several GWAPs have been developed with a number of different purposes Luca Galli
  • 9. 9The Fashion Trend Mining Pipeline Male, 24 Female, 22 Female?, ?? Luca Galli
  • 10. 10 Gwap Design: Sketchness!● Multiplayer Puzzle Game● Established genre: Guess and Draw (Pictionary, iSketch…)● Inversion Problem Mechanic● Players take turns into drawing the shapes of objects inside an image in order to make the other players guess the underlying object.● Two different roles: Sketcher Guesser● Objective: Garments tagging, body part segmentation, garment segmentation Luca Galli
  • 11. 11 Player Role: Sketcher Tag of the target object● The only player to see the low confidence image● “May” be asked to provide a tag for the image● Is asked to draw the contour of the object for which the tag is provided within the allotted time● Goal of the Sketcher is to let the other players guess the tag without providing any other hints than the contour Fashion image provided Contour of the garment by the pipeline provided by the player Luca Galli
  • 12. 12 Player Role: Guesser● Any other player in the game● His/Her goal is to guess the object for which the Sketcher has provided the contour● Not allowed to draw on the whiteboard, just to type in the chat box the probable answer as fast as possible● Scoring: ● Sketcher: 10 pts + 1 for each guesser ● Guesser: 10 pts to the first, then Contour of the garment decreasing down to five provided by the Sketcher Luca Galli
  • 13. 13 Task Injection: new approach to GWAPs● Traditional gwaps developed on ad-hoc basis and tailored for a specific task● Resulting gwaps may be not fun, unappealing to players or provide an experience that may still perceived by the users as work● The proposed GWAP exploits suitable modified game mechanics of a well known and appreciated category of games, called “drawing and guessing games” to implicitly solve segmentation problems while playing Luca Galli
  • 14. 14 Future Directions ● The proposed game concept is still under development and there are still several open questions and future improvements:● Task Injection: ● Does the task injection approach guarantee the same level of entertainment and participation for the players as the original instance of the game? ● How to measure entertainment anyhow?● Output Aggregation: ● How the results provided by different players on the same input be processed and grouped together in order to obtain a unique result? ● Majority voting not suitable for lines!● Integration within CUbRIK’s Gaming Framework: ● Exploiting the features provided by CUbRIK in order to manage players and their abilities and provide retention and reward features through gamification techniques. Luca Galli