An Analytical Study of Puzzle Selection Strategies for the ESP Game
Upcoming SlideShare
Loading in...5
×
 

An Analytical Study of Puzzle Selection Strategies for the ESP Game

on

  • 684 views

“Human Computation” represents a new paradigm of applications that take advantage of people’s desire to be entertained and produce useful metadata as a by-product. By creating games with a ...

“Human Computation” represents a new paradigm of applications that take advantage of people’s desire to be entertained and produce useful metadata as a by-product. By creating games with a purpose, human computation has shown promise in solving a variety of problems that computer computation cannot currently resolve completely. Using the ESP game as an example, we propose a metric, called system gain, for evaluating the performance of human computation systems, and also use analysis to study the properties of the ESP game. We argue that human computation systems should be played with a strategy. To this end, we implement an Optimal Puzzle Selection Strategy (OPSA) based on our analysis to improve human computation. Using a comprehensive set of simulations, we demonstrate that the proposed OPSA approach can effectively improve the system gain of the ESP game, as long as the number of puzzles in the system is sufficiently large.

Statistics

Views

Total Views
684
Views on SlideShare
672
Embed Views
12

Actions

Likes
0
Downloads
5
Comments
0

1 Embed 12

http://mmnet.iis.sinica.edu.tw 12

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

An Analytical Study of Puzzle Selection Strategies for the ESP Game An Analytical Study of Puzzle Selection Strategies for the ESP Game Presentation Transcript

  • An Analytical Study of Puzzle Selection Strategies for the ESP Game Ling-Jyh Chen, Bo-Chun Wang, Kuan-Ta Chen Academia Sinica Irwin King, and Jimmy Lee The Chinese University of Hong Kong
  • How the ESP Game Works? PLAYER 1 PLAYER 2 GUESSING: CAR GUESSING: BOY GUESSING: HAT GUESSING: CAR GUESSING: KID SUCCESS! SUCCESS! YOU AGREE ON CAR YOU AGREE ON CAR
  • Why is it Important? • Some statistics (July 2008) – 200,000+ players have contributed 50+ million labels. – Each player plays for a total of 91 minutes. – The throughput is about 233 labels/player/hour (i.e., one label every 15 seconds)
  • The Ideas behind the ESP Game • “Human Computation” represents a new paradigm of applications. – solve some problems which are difficult to be solved by computers • “Games With A Purpose” (GWAP) – by Dr. Luis von Ahn (CMU) – take advantage of people's desire to be entertained – produce useful metadata as a by-product
  • Our Contributions • We propose an evaluation metric for human computation systems. • We study the inner properties of the ESP game using analysis. • We propose an “Optimal Puzzle Selection Algorithm” (OPSA). • We present a comprehensive set of simulation results. • Our model is easy and applicable to other ESP-like systems.
  • The Main Ideas • There are two goals : – the system prefers to maximize the number of puzzles which have been played – the system prefers to take as many labels as possible for each puzzle • Unfortunately, there is a trade-off between the two goals.
  • The Proposed Metric System Gain ( G ) = ln ( N ) × ln ( S N ) the number of the puzzles thatsum ofbeen played labels the score average scores per puzzle the have all produced T T S r= ⇒N= = E[S] × r N N r r : each puzzle should be played T : total played rounds how many times in average r : each puzzle should be played E[S] : the expected score value of how many times in average each label
  • The Proposed Metric System Gain ( G ) = ln ( N ) × ln ( S N ) = l n (T r ) × l n ( E [ S ] × r ) l n ( T ) − l n ( E [ S ]) ⎞ 2 ⎛ = − ⎜ ln ( r ) − ⎟ +C ⎝ 2 ⎠ ln (T ) − ln ( E [ S ]) Maximum G = C , when r = e 2
  • The Proposed Metric Maximum G ln (T ) − ln ( E [ S ]) r=e 2
  • Puzzle Selection Algorithms • Optimal Puzzle Selection Algorithm (OPSA) – select a puzzle based on our analysis • Random Puzzle Selection Algorithm (RPSA) – select a puzzle by random • Fresh-first Puzzle Selection Algorithm (FPSA) – select a puzzle that has been played least frequently
  • Evaluation • Using Monte Carlo Simulations • # of rounds: T = 10,000 • # of puzzles: 100 <= M <= 5,000
  • Conclusion • We have proposed a metric to evaluate the performance of GWAP. • We argue that GWAP needs to be “played with a strategy”. • We propose the Optimal Puzzle Selection Algorithm (OPSA) for ESP-like games.
  • Thank You! http://nrl.iis.sinica.edu.tw