KissKissBan
A Competitive Human Computation
   Game for Image Annotation
Chien-Ju Ho*, Tao-Hsuan Chang+, Jong-Chuan Lee+,
...
Human Computation Game



Collaborative Mechanisms
Coalition Problem

Limited Data Diversity
The ESP Game

     • Coalition problem
       • partner with themselves
       • unified strategy
     • Limited diversity
...
Competitive Mechanism?
Player 1   Player 2




  DOG      CUTE
FLOWER     WOOD
 FLOOR      DOG
Player 1       Player 2




          DOG          CUTE
        FLOWER         WOOD
         FLOOR          DOG

DELIVER E...
Couple         Couple




          DOG          CUTE
        FLOWER         WOOD
         FLOOR          DOG

DELIVER EMO...
Blocker
Couple             Couple




         DOG
  DOG              CUTE
         WHITE
FLOWER             WOOD
 FLOOR  ...
Blocker
Couple                             Couple




                 DOG
  DOG                             CUTE
        ...
Blocker
  Couple                                    Couple




                       DOG                  CUTE
    DOG Bl...
Blocker
Couple                                   Couple




                    DOG                  CUTE
  DOG
          ...
Screenshots of KissKissBan
Intuitions Behind the Game
Incentive Structure

• Zero sum game
 • strict competitive
 • players can not benefit from coalition
• Different decision t...
Small-scale Experiment

• Image Source
 • 125 images randomly from ESP dataset
• Experiment Setup
 • 17-player gameplay su...
Gameplay Survey


• Fun
 • 3.76/5 overall rating
 • 15/17 players claim that would play again
 • over 60% like to play as ...
MTurk Experiment

• Collecting data from anonymous users
• Publish two games for comparison
 • KissKissBan
 • ESP Game wit...
Example of Collected Data
                   ESP    ML-KKB     BL-KKB
                          beach 3
                  ...
MTurk Experiment

• Data property
 • For each image
   • ESP: 39.96 labels, 6.56 distinct labels
   • KKB: 44.17 labels, 1...
Conclusion

• We proposed a competitive human
  computation game, KissKissBan.
• KissKissBan has two nice properties
 • a ...
More in the Future

• Publish the game on Facebook within the
  next one to two months.
• Study the different tagging beha...
Q &A
Upcoming SlideShare
Loading in...5
×

KissKissBan: A Competitive Human Computation Game for Image Annotation

1,139

Published on

In this paper, we propose a competitive human computation game, KissKissBan (KKB), for image annotation. KKB is different from other human computation games since it integrates both collaborative and competitive elements in the game design. In a KKB game, one player, the blocker, competes with the other two collaborative players, the couples; while the couples try to find consensual descriptions about an image, the blocker’s mission is to prevent the couples from reaching consensus. Because of its design, KKB possesses two nice properties over the traditional human computation game. First, since the blocker is encouraged to stop the couples from reaching consensual descriptions, he will try to detect and prevent coalition between the couples; therefore, these efforts naturally form a player-level cheating-proof mechanism. Second, to evade the restrictions set by the blocker, the couples would endeavor to bring up a more diverse set of image annotations. Experiments hosted on Amazon Mechanical Turk and a gameplay survey involving 17 participants have shown that KKB is a fun and efficient game for collecting diverse image annotations.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,139
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

KissKissBan: A Competitive Human Computation Game for Image Annotation

  1. 1. KissKissBan A Competitive Human Computation Game for Image Annotation Chien-Ju Ho*, Tao-Hsuan Chang+, Jong-Chuan Lee+, Jane Yun-jen Hsu+, Kuan-Ta Chen* *Academia Sinica +National Taiwan University
  2. 2. Human Computation Game Collaborative Mechanisms
  3. 3. Coalition Problem Limited Data Diversity
  4. 4. The ESP Game • Coalition problem • partner with themselves • unified strategy • Limited diversity • easy words first
  5. 5. Competitive Mechanism?
  6. 6. Player 1 Player 2 DOG CUTE FLOWER WOOD FLOOR DOG
  7. 7. Player 1 Player 2 DOG CUTE FLOWER WOOD FLOOR DOG DELIVER EMOTION BY TYPING THE SAME WORD
  8. 8. Couple Couple DOG CUTE FLOWER WOOD FLOOR DOG DELIVER EMOTION BY TYPING THE SAME WORD
  9. 9. Blocker Couple Couple DOG DOG CUTE WHITE FLOWER WOOD FLOOR DOG
  10. 10. Blocker Couple Couple DOG DOG CUTE WHITE FLOWER WOOD FLOOR DOG KissKissBan A Competitive Human Computation Game for Image Annotation
  11. 11. Blocker Couple Couple DOG CUTE DOG Blocked WHITE WOOD FLOWER FLOWER FLOOR The blocked words -5s game time -5s game time 1. First, the blocker has 7 seconds to type words 2. Then, the couples has 30 seconds to type words
  12. 12. Blocker Couple Couple DOG CUTE DOG WHITE WOOD FLOWER FLOWER FLOOR Couples win for matching within time limit 1. First, the blocker has 7 seconds to type words 2. Then, the couples has 30 seconds to type words
  13. 13. Screenshots of KissKissBan
  14. 14. Intuitions Behind the Game
  15. 15. Incentive Structure • Zero sum game • strict competitive • players can not benefit from coalition • Different decision time periods • harder words, longer consideration [Barry et al. 2001] • diverse data contribution
  16. 16. Small-scale Experiment • Image Source • 125 images randomly from ESP dataset • Experiment Setup • 17-player gameplay survey • Data collection on Mechanical Turk
  17. 17. Gameplay Survey • Fun • 3.76/5 overall rating • 15/17 players claim that would play again • over 60% like to play as a blocker
  18. 18. MTurk Experiment • Collecting data from anonymous users • Publish two games for comparison • KissKissBan • ESP Game without taboo words • Each game is played for ~5000 rounds
  19. 19. Example of Collected Data ESP ML-KKB BL-KKB beach 3 sea 9 water 3 man 21 man 8 sand 3 beach 10 ocean 3 sea 2 karate 5 black 1 ninja 1 water 1 china 1 kungfu 1 sand 1 ocean 1 ML-KKB: matching between couples BL-KKB: matching between couple and blocker
  20. 20. MTurk Experiment • Data property • For each image • ESP: 39.96 labels, 6.56 distinct labels • KKB: 44.17 labels, 11.53 distinct labels • A more diverse player contribution
  21. 21. Conclusion • We proposed a competitive human computation game, KissKissBan. • KissKissBan has two nice properties • a player-level anti-coalition mechanism • diverse player input
  22. 22. More in the Future • Publish the game on Facebook within the next one to two months. • Study the different tagging behaviors of the blocker and the couples. • Apply the competitive mechanism to more general human computation game design.
  23. 23. Q &A
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×