SlideShare a Scribd company logo
An Analytical Study of Puzzle
Selection Strategies for the ESP Game


      Ling-Jyh Chen, Bo-Chun Wang, Kuan-Ta Chen
                    Academia Sinica

               Irwin King, and Jimmy Lee
          The Chinese University of Hong Kong
How the ESP Game Works?
    PLAYER 1           PLAYER 2




 GUESSING: CAR       GUESSING: BOY

  GUESSING: HAT     GUESSING: CAR

  GUESSING: KID

 SUCCESS!           SUCCESS!
 YOU AGREE ON CAR   YOU AGREE ON CAR
Why is it Important?
• Some statistics (July 2008)
  – 200,000+ players have contributed 50+ million
    labels.

  – Each player plays for a total of 91 minutes.

  – The throughput is about 233
    labels/player/hour (i.e., one label every 15
    seconds)
The Ideas behind the ESP Game
• “Human Computation” represents a new
  paradigm of applications.
  – solve some problems which are difficult to be solved
    by computers


• “Games With A Purpose” (GWAP) – by Dr.
  Luis von Ahn (CMU)
  – take advantage of people's desire to be entertained
  – produce useful metadata as a by-product
Our Contributions
• We propose an evaluation metric for human computation
  systems.

• We study the inner properties of the ESP game using analysis.

• We propose an “Optimal Puzzle Selection Algorithm” (OPSA).

• We present a comprehensive set of simulation results.

• Our model is easy and applicable to other ESP-like systems.
The Main Ideas
• There are two goals :
  – the system prefers to maximize the number of
    puzzles which have been played
  – the system prefers to take as many labels as
    possible for each puzzle


• Unfortunately, there is a trade-off between
  the two goals.
The Proposed Metric

                System Gain ( G )
                = ln ( N ) × ln ( S N )
    the number of the puzzles thatsum ofbeen played labels
                       the score average scores per puzzle
                               the have all produced




       T    T                                 S
     r= ⇒N=                                     = E[S] × r
                                              N
       N    r                        r : each puzzle should be played
      T : total played rounds            how many times in average
r : each puzzle should be played     E[S] : the expected score value of
    how many times in average                     each label
The Proposed Metric
System Gain ( G )
= ln ( N ) × ln ( S N )
= l n (T r ) × l n ( E [ S ] × r )
                 l n ( T ) − l n ( E [ S ]) ⎞
                                                        2
    ⎛
= − ⎜ ln ( r ) −                            ⎟ +C
    ⎝                        2              ⎠
                                     ln (T ) − ln ( E [ S ])
     Maximum G = C , when r = e                 2
The Proposed Metric

Maximum G




                  ln (T ) − ln ( E [ S ])
            r=e              2
Puzzle Selection Algorithms
• Optimal Puzzle Selection Algorithm (OPSA)
  – select a puzzle based on our analysis

• Random Puzzle Selection Algorithm (RPSA)
  – select a puzzle by random

• Fresh-first Puzzle Selection Algorithm (FPSA)
   – select a puzzle that has been played least
     frequently
Evaluation
• Using Monte Carlo Simulations
• # of rounds: T = 10,000
• # of puzzles: 100 <= M <= 5,000
Conclusion
• We have proposed a metric to evaluate
  the performance of GWAP.

• We argue that GWAP needs to be “played
  with a strategy”.

• We propose the Optimal Puzzle Selection
  Algorithm (OPSA) for ESP-like games.
Thank You!
http://nrl.iis.sinica.edu.tw

More Related Content

Similar to An Analytical Study of Puzzle Selection Strategies for the ESP Game

Finalver
FinalverFinalver
Finalver
Natan Katz
 
AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Sho...
AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Sho...AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Sho...
AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Sho...
Joonhyung Lee
 
AlphaGo Zero: Mastering the Game of Go Without Human Knowledge
AlphaGo Zero: Mastering the Game of Go Without Human KnowledgeAlphaGo Zero: Mastering the Game of Go Without Human Knowledge
AlphaGo Zero: Mastering the Game of Go Without Human Knowledge
Joonhyung Lee
 
Regression
RegressionRegression
Regression
Ncib Lotfi
 
chapter1.pdf ......................................
chapter1.pdf ......................................chapter1.pdf ......................................
chapter1.pdf ......................................
nourhandardeer3
 
Applied Algorithms and Structures week999
Applied Algorithms and Structures week999Applied Algorithms and Structures week999
Applied Algorithms and Structures week999
fashiontrendzz20
 
Introduction to Image Processing
Introduction to Image ProcessingIntroduction to Image Processing
Introduction to Image Processing
Israel Gbati
 
Unit II B - Game Theory
Unit II B - Game TheoryUnit II B - Game Theory
Efficient Analysis of high-dimensional data in tensor formats
Efficient Analysis of high-dimensional data in tensor formatsEfficient Analysis of high-dimensional data in tensor formats
Efficient Analysis of high-dimensional data in tensor formats
Alexander Litvinenko
 
Clustering:k-means, expect-maximization and gaussian mixture model
Clustering:k-means, expect-maximization and gaussian mixture modelClustering:k-means, expect-maximization and gaussian mixture model
Clustering:k-means, expect-maximization and gaussian mixture model
jins0618
 
L4
L4L4
Introduction to Recursion (Python)
Introduction to Recursion (Python)Introduction to Recursion (Python)
Introduction to Recursion (Python)
Thai Pangsakulyanont
 
Robust Price of Anarchy
Robust Price of AnarchyRobust Price of Anarchy
Robust Price of Anarchy
鴻鈞 王
 
Machine Learning Chapter 11 2
Machine Learning Chapter 11 2Machine Learning Chapter 11 2
Machine Learning Chapter 11 2
butest
 
cai
caicai
Introduction to Alphago Zero
Introduction to Alphago ZeroIntroduction to Alphago Zero
Introduction to Alphago Zero
Chia-Ching Lin
 
Replica exchange MCMC
Replica exchange MCMCReplica exchange MCMC
Replica exchange MCMC
. .
 
Algorithms - "Chapter 2 getting started"
Algorithms - "Chapter 2 getting started"Algorithms - "Chapter 2 getting started"
Algorithms - "Chapter 2 getting started"
Ra'Fat Al-Msie'deen
 
Section9 stochastic
Section9 stochasticSection9 stochastic
Section9 stochastic
cairo university
 
PyCon2009_AI_Alt
PyCon2009_AI_AltPyCon2009_AI_Alt
PyCon2009_AI_Alt
Hiroshi Ono
 

Similar to An Analytical Study of Puzzle Selection Strategies for the ESP Game (20)

Finalver
FinalverFinalver
Finalver
 
AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Sho...
AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Sho...AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Sho...
AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Sho...
 
AlphaGo Zero: Mastering the Game of Go Without Human Knowledge
AlphaGo Zero: Mastering the Game of Go Without Human KnowledgeAlphaGo Zero: Mastering the Game of Go Without Human Knowledge
AlphaGo Zero: Mastering the Game of Go Without Human Knowledge
 
Regression
RegressionRegression
Regression
 
chapter1.pdf ......................................
chapter1.pdf ......................................chapter1.pdf ......................................
chapter1.pdf ......................................
 
Applied Algorithms and Structures week999
Applied Algorithms and Structures week999Applied Algorithms and Structures week999
Applied Algorithms and Structures week999
 
Introduction to Image Processing
Introduction to Image ProcessingIntroduction to Image Processing
Introduction to Image Processing
 
Unit II B - Game Theory
Unit II B - Game TheoryUnit II B - Game Theory
Unit II B - Game Theory
 
Efficient Analysis of high-dimensional data in tensor formats
Efficient Analysis of high-dimensional data in tensor formatsEfficient Analysis of high-dimensional data in tensor formats
Efficient Analysis of high-dimensional data in tensor formats
 
Clustering:k-means, expect-maximization and gaussian mixture model
Clustering:k-means, expect-maximization and gaussian mixture modelClustering:k-means, expect-maximization and gaussian mixture model
Clustering:k-means, expect-maximization and gaussian mixture model
 
L4
L4L4
L4
 
Introduction to Recursion (Python)
Introduction to Recursion (Python)Introduction to Recursion (Python)
Introduction to Recursion (Python)
 
Robust Price of Anarchy
Robust Price of AnarchyRobust Price of Anarchy
Robust Price of Anarchy
 
Machine Learning Chapter 11 2
Machine Learning Chapter 11 2Machine Learning Chapter 11 2
Machine Learning Chapter 11 2
 
cai
caicai
cai
 
Introduction to Alphago Zero
Introduction to Alphago ZeroIntroduction to Alphago Zero
Introduction to Alphago Zero
 
Replica exchange MCMC
Replica exchange MCMCReplica exchange MCMC
Replica exchange MCMC
 
Algorithms - "Chapter 2 getting started"
Algorithms - "Chapter 2 getting started"Algorithms - "Chapter 2 getting started"
Algorithms - "Chapter 2 getting started"
 
Section9 stochastic
Section9 stochasticSection9 stochastic
Section9 stochastic
 
PyCon2009_AI_Alt
PyCon2009_AI_AltPyCon2009_AI_Alt
PyCon2009_AI_Alt
 

More from Academia Sinica

Computational Social Science:The Collaborative Futures of Big Data, Computer ...
Computational Social Science:The Collaborative Futures of Big Data, Computer ...Computational Social Science:The Collaborative Futures of Big Data, Computer ...
Computational Social Science:The Collaborative Futures of Big Data, Computer ...
Academia Sinica
 
Games on Demand: Are We There Yet?
Games on Demand: Are We There Yet?Games on Demand: Are We There Yet?
Games on Demand: Are We There Yet?
Academia Sinica
 
Detecting In-Situ Identity Fraud on Social Network Services: A Case Study on ...
Detecting In-Situ Identity Fraud on Social Network Services: A Case Study on ...Detecting In-Situ Identity Fraud on Social Network Services: A Case Study on ...
Detecting In-Situ Identity Fraud on Social Network Services: A Case Study on ...
Academia Sinica
 
Cloud Gaming Onward: Research Opportunities and Outlook
Cloud Gaming Onward: Research Opportunities and OutlookCloud Gaming Onward: Research Opportunities and Outlook
Cloud Gaming Onward: Research Opportunities and Outlook
Academia Sinica
 
Quantifying User Satisfaction in Mobile Cloud Games
Quantifying User Satisfaction in Mobile Cloud GamesQuantifying User Satisfaction in Mobile Cloud Games
Quantifying User Satisfaction in Mobile Cloud Games
Academia Sinica
 
量化「樂趣」-以心理生理量測探究數位娛樂商品之市場價值
量化「樂趣」-以心理生理量測探究數位娛樂商品之市場價值量化「樂趣」-以心理生理量測探究數位娛樂商品之市場價值
量化「樂趣」-以心理生理量測探究數位娛樂商品之市場價值Academia Sinica
 
On The Battle between Online Gamers and Lags
On The Battle between Online Gamers and LagsOn The Battle between Online Gamers and Lags
On The Battle between Online Gamers and Lags
Academia Sinica
 
Understanding The Performance of Thin-Client Gaming
Understanding The Performance of Thin-Client GamingUnderstanding The Performance of Thin-Client Gaming
Understanding The Performance of Thin-Client Gaming
Academia Sinica
 
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework
Quantifying QoS Requirements of Network Services: A Cheat-Proof FrameworkQuantifying QoS Requirements of Network Services: A Cheat-Proof Framework
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework
Academia Sinica
 
Online Game QoE Evaluation using Paired Comparisons
Online Game QoE Evaluation using Paired ComparisonsOnline Game QoE Evaluation using Paired Comparisons
Online Game QoE Evaluation using Paired Comparisons
Academia Sinica
 
GamingAnywhere: An Open Cloud Gaming System
GamingAnywhere: An Open Cloud Gaming SystemGamingAnywhere: An Open Cloud Gaming System
GamingAnywhere: An Open Cloud Gaming System
Academia Sinica
 
Are All Games Equally Cloud-Gaming-Friendly? An Electromyographic Approach
Are All Games Equally Cloud-Gaming-Friendly? An Electromyographic ApproachAre All Games Equally Cloud-Gaming-Friendly? An Electromyographic Approach
Are All Games Equally Cloud-Gaming-Friendly? An Electromyographic Approach
Academia Sinica
 
Forecasting Online Game Addictiveness
Forecasting Online Game AddictivenessForecasting Online Game Addictiveness
Forecasting Online Game Addictiveness
Academia Sinica
 
Identifying MMORPG Bots: A Traffic Analysis Approach
Identifying MMORPG Bots: A Traffic Analysis ApproachIdentifying MMORPG Bots: A Traffic Analysis Approach
Identifying MMORPG Bots: A Traffic Analysis Approach
Academia Sinica
 
Toward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes
Toward an Understanding of the Processing Delay of Peer-to-Peer Relay NodesToward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes
Toward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes
Academia Sinica
 
Inferring Speech Activity from Encrypted Skype Traffic
Inferring Speech Activity from Encrypted Skype TrafficInferring Speech Activity from Encrypted Skype Traffic
Inferring Speech Activity from Encrypted Skype Traffic
Academia Sinica
 
Game Bot Detection Based on Avatar Trajectory
Game Bot Detection Based on Avatar TrajectoryGame Bot Detection Based on Avatar Trajectory
Game Bot Detection Based on Avatar Trajectory
Academia Sinica
 
Improving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
Improving Reliability of Web 2.0-based Rating Systems Using Per-user TrustinessImproving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
Improving Reliability of Web 2.0-based Rating Systems Using Per-user TrustinessAcademia Sinica
 
A Collusion-Resistant Automation Scheme for Social Moderation Systems
A Collusion-Resistant Automation Scheme for Social Moderation SystemsA Collusion-Resistant Automation Scheme for Social Moderation Systems
A Collusion-Resistant Automation Scheme for Social Moderation Systems
Academia Sinica
 
Tuning Skype’s Redundancy Control Algorithm for User Satisfaction
Tuning Skype’s Redundancy Control Algorithm for User SatisfactionTuning Skype’s Redundancy Control Algorithm for User Satisfaction
Tuning Skype’s Redundancy Control Algorithm for User Satisfaction
Academia Sinica
 

More from Academia Sinica (20)

Computational Social Science:The Collaborative Futures of Big Data, Computer ...
Computational Social Science:The Collaborative Futures of Big Data, Computer ...Computational Social Science:The Collaborative Futures of Big Data, Computer ...
Computational Social Science:The Collaborative Futures of Big Data, Computer ...
 
Games on Demand: Are We There Yet?
Games on Demand: Are We There Yet?Games on Demand: Are We There Yet?
Games on Demand: Are We There Yet?
 
Detecting In-Situ Identity Fraud on Social Network Services: A Case Study on ...
Detecting In-Situ Identity Fraud on Social Network Services: A Case Study on ...Detecting In-Situ Identity Fraud on Social Network Services: A Case Study on ...
Detecting In-Situ Identity Fraud on Social Network Services: A Case Study on ...
 
Cloud Gaming Onward: Research Opportunities and Outlook
Cloud Gaming Onward: Research Opportunities and OutlookCloud Gaming Onward: Research Opportunities and Outlook
Cloud Gaming Onward: Research Opportunities and Outlook
 
Quantifying User Satisfaction in Mobile Cloud Games
Quantifying User Satisfaction in Mobile Cloud GamesQuantifying User Satisfaction in Mobile Cloud Games
Quantifying User Satisfaction in Mobile Cloud Games
 
量化「樂趣」-以心理生理量測探究數位娛樂商品之市場價值
量化「樂趣」-以心理生理量測探究數位娛樂商品之市場價值量化「樂趣」-以心理生理量測探究數位娛樂商品之市場價值
量化「樂趣」-以心理生理量測探究數位娛樂商品之市場價值
 
On The Battle between Online Gamers and Lags
On The Battle between Online Gamers and LagsOn The Battle between Online Gamers and Lags
On The Battle between Online Gamers and Lags
 
Understanding The Performance of Thin-Client Gaming
Understanding The Performance of Thin-Client GamingUnderstanding The Performance of Thin-Client Gaming
Understanding The Performance of Thin-Client Gaming
 
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework
Quantifying QoS Requirements of Network Services: A Cheat-Proof FrameworkQuantifying QoS Requirements of Network Services: A Cheat-Proof Framework
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework
 
Online Game QoE Evaluation using Paired Comparisons
Online Game QoE Evaluation using Paired ComparisonsOnline Game QoE Evaluation using Paired Comparisons
Online Game QoE Evaluation using Paired Comparisons
 
GamingAnywhere: An Open Cloud Gaming System
GamingAnywhere: An Open Cloud Gaming SystemGamingAnywhere: An Open Cloud Gaming System
GamingAnywhere: An Open Cloud Gaming System
 
Are All Games Equally Cloud-Gaming-Friendly? An Electromyographic Approach
Are All Games Equally Cloud-Gaming-Friendly? An Electromyographic ApproachAre All Games Equally Cloud-Gaming-Friendly? An Electromyographic Approach
Are All Games Equally Cloud-Gaming-Friendly? An Electromyographic Approach
 
Forecasting Online Game Addictiveness
Forecasting Online Game AddictivenessForecasting Online Game Addictiveness
Forecasting Online Game Addictiveness
 
Identifying MMORPG Bots: A Traffic Analysis Approach
Identifying MMORPG Bots: A Traffic Analysis ApproachIdentifying MMORPG Bots: A Traffic Analysis Approach
Identifying MMORPG Bots: A Traffic Analysis Approach
 
Toward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes
Toward an Understanding of the Processing Delay of Peer-to-Peer Relay NodesToward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes
Toward an Understanding of the Processing Delay of Peer-to-Peer Relay Nodes
 
Inferring Speech Activity from Encrypted Skype Traffic
Inferring Speech Activity from Encrypted Skype TrafficInferring Speech Activity from Encrypted Skype Traffic
Inferring Speech Activity from Encrypted Skype Traffic
 
Game Bot Detection Based on Avatar Trajectory
Game Bot Detection Based on Avatar TrajectoryGame Bot Detection Based on Avatar Trajectory
Game Bot Detection Based on Avatar Trajectory
 
Improving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
Improving Reliability of Web 2.0-based Rating Systems Using Per-user TrustinessImproving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
Improving Reliability of Web 2.0-based Rating Systems Using Per-user Trustiness
 
A Collusion-Resistant Automation Scheme for Social Moderation Systems
A Collusion-Resistant Automation Scheme for Social Moderation SystemsA Collusion-Resistant Automation Scheme for Social Moderation Systems
A Collusion-Resistant Automation Scheme for Social Moderation Systems
 
Tuning Skype’s Redundancy Control Algorithm for User Satisfaction
Tuning Skype’s Redundancy Control Algorithm for User SatisfactionTuning Skype’s Redundancy Control Algorithm for User Satisfaction
Tuning Skype’s Redundancy Control Algorithm for User Satisfaction
 

Recently uploaded

Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
ScyllaDB
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
FilipTomaszewski5
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
ScyllaDB
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
Fwdays
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Neo4j
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
UiPathCommunity
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 

Recently uploaded (20)

Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 

An Analytical Study of Puzzle Selection Strategies for the ESP Game

  • 1. An Analytical Study of Puzzle Selection Strategies for the ESP Game Ling-Jyh Chen, Bo-Chun Wang, Kuan-Ta Chen Academia Sinica Irwin King, and Jimmy Lee The Chinese University of Hong Kong
  • 2. How the ESP Game Works? PLAYER 1 PLAYER 2 GUESSING: CAR GUESSING: BOY GUESSING: HAT GUESSING: CAR GUESSING: KID SUCCESS! SUCCESS! YOU AGREE ON CAR YOU AGREE ON CAR
  • 3. Why is it Important? • Some statistics (July 2008) – 200,000+ players have contributed 50+ million labels. – Each player plays for a total of 91 minutes. – The throughput is about 233 labels/player/hour (i.e., one label every 15 seconds)
  • 4. The Ideas behind the ESP Game • “Human Computation” represents a new paradigm of applications. – solve some problems which are difficult to be solved by computers • “Games With A Purpose” (GWAP) – by Dr. Luis von Ahn (CMU) – take advantage of people's desire to be entertained – produce useful metadata as a by-product
  • 5. Our Contributions • We propose an evaluation metric for human computation systems. • We study the inner properties of the ESP game using analysis. • We propose an “Optimal Puzzle Selection Algorithm” (OPSA). • We present a comprehensive set of simulation results. • Our model is easy and applicable to other ESP-like systems.
  • 6. The Main Ideas • There are two goals : – the system prefers to maximize the number of puzzles which have been played – the system prefers to take as many labels as possible for each puzzle • Unfortunately, there is a trade-off between the two goals.
  • 7. The Proposed Metric System Gain ( G ) = ln ( N ) × ln ( S N ) the number of the puzzles thatsum ofbeen played labels the score average scores per puzzle the have all produced T T S r= ⇒N= = E[S] × r N N r r : each puzzle should be played T : total played rounds how many times in average r : each puzzle should be played E[S] : the expected score value of how many times in average each label
  • 8. The Proposed Metric System Gain ( G ) = ln ( N ) × ln ( S N ) = l n (T r ) × l n ( E [ S ] × r ) l n ( T ) − l n ( E [ S ]) ⎞ 2 ⎛ = − ⎜ ln ( r ) − ⎟ +C ⎝ 2 ⎠ ln (T ) − ln ( E [ S ]) Maximum G = C , when r = e 2
  • 9. The Proposed Metric Maximum G ln (T ) − ln ( E [ S ]) r=e 2
  • 10. Puzzle Selection Algorithms • Optimal Puzzle Selection Algorithm (OPSA) – select a puzzle based on our analysis • Random Puzzle Selection Algorithm (RPSA) – select a puzzle by random • Fresh-first Puzzle Selection Algorithm (FPSA) – select a puzzle that has been played least frequently
  • 11. Evaluation • Using Monte Carlo Simulations • # of rounds: T = 10,000 • # of puzzles: 100 <= M <= 5,000
  • 12. Conclusion • We have proposed a metric to evaluate the performance of GWAP. • We argue that GWAP needs to be “played with a strategy”. • We propose the Optimal Puzzle Selection Algorithm (OPSA) for ESP-like games.