Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Matching Game Mechanics and Human Computation Tasks in Games with a Purpose

2,304 views

Published on

Presentation held at ACM Workshop on Serious Games at ACM Multimedia 2014 to introduce techniques that can be used to match Human Computation Tasks to Game Mechanics.
The development process of a traditional digital game is described to put the basis on which the development of GWAP can be described. The concept of Human Computation Task is provided along with the most common multimedia refinement tasks that can be found in literature and everyday scenarios. Finally game mechanics that could be matched to particular task instances are shown, along with examples. The entire process has been applied to a real Game with a Purpose scenario.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Matching Game Mechanics and Human Computation Tasks in Games with a Purpose

  1. 1. MATCHING GAME MECHANICS AND HUMAN COMPUTATION TASKS IN GAMES WITH A PURPOSE LUCA GALLI POLITECNICO DI MILANO ACM International Workshop on Serious Games
  2. 2. ABOUT MYSELF LUCA GALLI Ph.D. Student - Politecnico di Milano lgalli@elet.polimi.it 2 • RESEARCH BACKGROUND AND INTERESTS • Crowdsourcing and Human Computation • Game Design • Robotics & AI ACM International Workshop on Serious Games
  3. 3. OUTLINE • Brief recap on Games with a Purpose • Current Issues • Development Process • Task Design • Game Mechanics & Operation Matching • Use Case Study: Sketchness ACM International Workshop on Serious Games 3
  4. 4. GAMES WITH A PURPOSE ACM International Workshop on Serious Games 4 Games with a Purpose (GWAPs) are digital games that generate useful data as a by-product of play. [vA06,LvA09] The design of a GWAP requires to create a game so that its structure encourages computation, correctness of the output and players retention.
  5. 5. ARE THEY WORKING? ACM International Workshop on Serious Games 5 ESP Game Purposes: Let players determine the contents of images by submitting meaningful labels they can agree on. Results: A total of 13,630 people played the game during the test phase, generating 1,271,451 labels for 293,760 different images. Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game.
  6. 6. GAME… MECHANICS? ACM International Workshop on Serious Games 6 Input Agreement Output Agreement Inversion problem “These game mechanics are the building blocks with which one could create games with a purpose to solve computational tasks”. Designing Games with a Purpose - Luis Von Ahn
  7. 7. NO! THEY ARE VALIDATION TEMPLATES! ACM International Workshop on Serious Games 7
  8. 8. ACM International Workshop on Serious Games 8 Katharina Siorpaes and Martin Hepp. 2008. OntoGame: weaving the semantic web by online games.
  9. 9. GAMES WITH A PURPOSE: CURRENT ISSUES ACM International Workshop on Serious Games 9 ● “Ad-Hoc” Design ● Lacking of mapping guidelines from task to game mechanics E.G. Real user comments about OntoGalaxy
  10. 10. ISSUE 1: «AD HOC» DESIGN ACM International Workshop on Serious Games 10 Game design is primarily an artistic process and reliance on formal procedures is inimical to creativity Yet a GWAP have a lot in common with traditional software due to their very nature…
  11. 11. GAME DEVELOPMENT PROCESS ACM International Workshop on Serious Games 11 Derived by modeling the guidelines and best practices found in: Fullerton, T.; Swain, C. & Hoffman, S. Game Design Workshop: A playcentric approach to creating innovative games, 2008 Crowford, C. The Art of Computer Game Design, 1984
  12. 12. 12 Taken from the Book: Level Up, By Scott Rogers ACM International Workshop on Serious Games
  13. 13. ACM International Workshop on Serious Games 13 Taken from the book: Game Mechanics: Joris Dorman, Ernest Adams
  14. 14. ACM International Workshop on Serious Games 14 Board Prototype for IGAM Challenge
  15. 15. ACM International Workshop on Serious Games 15 Prototype, Alex Mercer
  16. 16. ACM International Workshop on Serious Games 16 Alpha version of Unreal Tournament
  17. 17. GWAP DEVELOPMENT PROCESS ACM International Workshop on Serious Games 17
  18. 18. TASK DESIGN A Human Computation Task (HCT) is a “unit of work“ assigned to a user of a Human Computation system. Input Objects ACM International Workshop on Serious Games 18 Task Task Description Output Objects Execution Interface Operations Aggregation Strategy
  19. 19. TASK DESIGN: INPUT OBJECT Input object for a HC task may include texts, images, audio segments, video segments, other unstructured data, structured data Input Objects ACM International Workshop on Serious Games 19 Task Task Description Output Objects Operations Aggregation Strategy
  20. 20. TASK DESIGN: DESCRIPTION A description details the goal that has to be reached and a set of admissible operations that represent the mean with which this particular task can be accomplished by the user. Input Objects ACM International Workshop on Serious Games 20 Task Task Description Output Objects Operations Aggregation Strategy Execution Interface
  21. 21. TASK DESIGN: AGGREGATION Usually one single contribution from a user does not suffice for obtaining meaningful results, thus the contribution of several users is aggregated using techniques such as ranking, clustering, majority voting… Input Objects ACM International Workshop on Serious Games 21 Task Task Description Output Objects Operations Aggregation Strategy Execution Interface
  22. 22. DEFINING OPERATIONS Input Objects ACM International Workshop on Serious Games 22 Execution Interface Task Output Objects Operations Task Description Aggregation Strategy
  23. 23. OPERATION TYPES ACM International Workshop on Serious Games 23 They fall in two broad categories: Generative tasks Decision tasks The media refining tasks that are suited to benefit from human contributions are: • Object Recognition/Identification/Detection • Clustering • Ordering • Natural Language Processing • State Space Exploration • Content Generation/Submission • User preference/opinion elicitation
  24. 24. OBJECT RECOGNITION IDENTIFICATION, DETECTION ACM International Workshop on Serious Games 24 Given a specific object, identify it in the image or environment with an annotation which selects a subset of samples with a particular meaning. (E.G. Tracing a bounding box)
  25. 25. CLUSTERING ACM International Workshop on Serious Games 25 Define a (subjective) similarity measure to compare the input data with and group objects into clusters based on it.
  26. 26. ORDERING ACM International Workshop on Serious Games 26 Define a (subjective) evaluation criteria to compare the input data and order the objects based on the chosen criteria.
  27. 27. NATURAL LANGUAGE PROCESSING ACM International Workshop on Serious Games 27 Solving various Natural Language Processing tasks by exploiting humans' ability to understand natural language And so on…
  28. 28. GWAP DEVELOPMENT PROCESS ACM International Workshop on Serious Games 28
  29. 29. «ISSUE 2»: TASK TO GAME MECHANICS MATCHING How to match the task to proper game mechanics in order to design a GWAP? ACM International Workshop on Serious Games 29 Does a game with game mechanics similar to the task exists? If so, integrate the task within the game If not, custom game mechanics has to be implemented.
  30. 30. SOLUTION MECHANICS Game mechanics that were similar to the proposed multimedia refinement tasks has been defined from the analysis of manuals about game design and specialized websites such as BoardGameGeek Agreement Betting/Wagering Memory ACM International Workshop on Serious Games 30 Trivia Tile (Resource) - Placement Pattern Building Line Drawing Bluffing Area Enclosure
  31. 31. AGREEMENT ACM International Workshop on Serious Games 31 Players are requested to reach an agreement over a question or a topic based on some hints provided by the game. Agreement is one of the most widely used mechanics in GWAP, being the foundation on which templates like input-agreement and output-agreement rely on to be able to automatically validate the contributions of different players.
  32. 32. AGREEMENT APPLIED TO GWAPS ACM International Workshop on Serious Games 32 Purpose: Collect “common-sense” fact for specific words. Game Mechanic: players must agree on the same tags by relying on an image as their only common hint
  33. 33. TILE PLACEMENT ACM International Workshop on Serious Games 33 Tile Placement games feature placing a piece to score points, based on adjacent pieces or pieces in the same group/cluster, taking into consideration non-spatial properties like color, "feature completion", cluster size etc.
  34. 34. TILE PLACEMENT APPLIED TO GWAPS Game Mechanic: align the sequence contained in each row in order to obtain the greatest number of columns with matching colors. ACM International Workshop on Serious Games 34 Purpose: Used to arrange the sequences of D.N.A, R.N.A or proteins to identify regions of similarity Kawrykow A, Roumanis G, Kam A, Kwak D, Leung C, et al. (2012) Phylo: A Citizen Science Approach for Improving Multiple Sequence Alignment
  35. 35. MEMORY ACM International Workshop on Serious Games 35 Games that use the Memory mechanic require players to recall previous game events or information in order to reach an objective.
  36. 36. MEMORY APPLIED TO GWAPS ACM International Workshop on Serious Games 36 Purpose: Find similar Images Game Mechanic: Exploit the visual memory of players to find images which are similar. And so on…
  37. 37. GAME TO MECHANICS MATCHING ACM International Workshop on Serious Games 37 Game Mechanics Task Type Significative Examples Agreement Object Recognition, Clustering, Ordering, NLP ESP Game, TagATune Tile (Resource)- Placement Clustering, Ordering Phylo Line Drawing Object Recognition Sketchness, Squigl Betting/Wagering User Preferences / Opinion Elicitation N/A Memory Clustering FlipIt Pattern Building State Exploration FoldIt, Eyewire Bluffing Ordering, Object Identification Disguise, Search War Trivia Natural Language Processing Verbosity, WebPardy Area Enclosure Object Recognition PeekABoom, Ask’nSeek
  38. 38. CASE STUDY: FASHION TREND MINING Problem statement: segment fashion images for mining trends based on visual features of garments (e.g. color and texture) Use case: identifying trends in collections of images of people and garments Applications: retrieving similar garments, inspect clothing trends in image collections, analyzing trends change in the years ACM International Workshop on Serious Games 38 Color descriptors Texture descriptors coarse (sub-)image similarity
  39. 39. THE FASHION TREND MINING PIPELINE ACM International Workshop on Serious Games 39 Male, 24 Female, 22Female?, ??
  40. 40. REQUIREMENTS SPECIFICATION Input Objects: A fashion image, an optional tag defining the garment to identify. Output Data: For each submitted task the game has to provide the contour of the garment within the image (Polyline) and a tag defining the garment that has been segmented ACM International Workshop on Serious Games 40 Task Description: Recognize if a particular garment is present within a picture or define a new one and outline its contours. Aggregation Strategy: assign a value of 0 to each pixel outside the contour and 1 to each pixel contained within the contour, sum all the contribution and apply a threshold based on the number of players.
  41. 41. LINE DRAWING ACM International Workshop on Serious Games 41 Games that make use of this mechanics involve drawing drawing of lines in one way or another.
  42. 42. LINE DRAWING APPLIED TO GWAPS ACM International Workshop on Serious Games 42 Purpose: Segment images Game Mechanic: draw the shapes of objects in a provided image in order to make the other players guess the underlying object.
  43. 43. GWAP DESIGN Solution Mechanics: Line Drawing Agreement Established genre: Draw and Guess Inversion Problem Mechanic PLAYERS:Number >=2 Multilateral Competition Two different roles: Sketcher: has to draw the contours of the stated garment Guesser: has to guess the garment drawn by the sketcher ACM International Workshop on Serious Games 43
  44. 44. PLAYER ROLE: SKETCHER ACM International Workshop on Serious Games 44 ● The only player to see the low confidence image ● “May” be asked to provide a tag for the image ● Is asked to draw the contour of the object for which the tag is provided within the allotted time ● Goal of the Sketcher is to let the other players guess the tag within a time slot without providing any other hints than the contour
  45. 45. PLAYER ROLE: GUESSER ACM International Workshop on Serious Games 45 ● Any other player in the game ● His/Her goal is to guess the object for which the Sketcher has provided the contour ● Not allowed to draw on the whiteboard, just to type in the chat box the probable answer as fast as possible ● Scoring: ● Sketcher: 10 pts + 1 for each guesser ● Guesser: 10 pts to the first, then decreasing down to five
  46. 46. RESULTS: ACM International Workshop on Serious Games
  47. 47. RESULTS: PIXEL ACCURACY u: 95.4 std: 1.3 (whole dataset) u: 89.0 std:0.8 Sketchness Aggregator Kota Yamaguchi, M Hadi Kiapour, Luis E Ortiz, Tamara L Berg, "Parsing Clothing in Fashion Photographs", CVPR 2012 ACM International Workshop on Serious Games
  48. 48. BUT CONTROLLED DATASET! ACM International Workshop on Serious Games
  49. 49. RESULTS: USER SUBMITTED CONTENT Baseline: Background Algorithm Mean Standard Deviation Fashionista 81.7901 13.555 Paperdoll 78.2486 13.5549 Sketchness 93.2669 9.6266 Baseline: against manually annotated groundtruth Algorithm Mean Standard Deviation Fashionista 26.7811 31.4905 Paperdoll 54.2095 37.056 Sketchness 81.3803 24.4377 ACM International Workshop on Serious Games
  50. 50. THE CUBRIK PROJECT 50 ● CUbRIK is a research project financed by the European Union ● Goals: ● Advance the architecture of multimedia search ● Exploit the human contribution in multimedia search ● Use open-source components provided by the community ● Start up a search business ecosystem ● http://www.cubrikproject.eu/ ACM International Workshop on Serious Games

×