Systematic Analysis ofOutput Agreement Games:Effects of Gaming Environment,Social Interaction, and FeedbackShih-Wen Huang ...
GWAP Today – ESP Game    [Von Ahn and Dabbish, CHI 2004]
GWAP Today - Foldit   [Cooper et al., Nature 2010]
Two Unanswered Questions1. Can GWAP collect outputs with high quality?   Why?2. Which component of GWAP motivate   volunte...
Output Agreement Games
Output Agreement Games                          Database1. Randomly matchmultiple strangersand provide them withthe same i...
Output Agreement Games                              Database1. Randomly matchmultiple strangersand provide them withthe sa...
Output Agreement Games                              Database1. Randomly match                            3. Players will b...
Three Major Components• Gaming Environment (G): Encouraging players  to generate matched answers using scoring  system, le...
Five Different Interfaces•   (B)•   (F)•   (F+S)•   (F+G)•   (F+S+G)B: baseline F: feedback G: gaming environmentS: social...
Five Different Interfaces•   (B)•   (F)•   (F+S)•   (F+G)•   (F+S+G)B: baseline F: feedback G: gaming environmentS: social...
Five Different Interfaces•   (B)•   (F)•   (F+S)•   (F+G)•   (F+S+G)    Teammate searching page   Teammate waiting pageB: ...
Five Different Interfaces•   (B)•   (F)•   (F+S)•   (F+G)•   (F+S+G)B: baseline F: feedback G: gaming environmentS: social...
Five Different Interfaces•   (B)•   (F)•   (F+S)•   (F+G)•   (F+S+G)B: baseline F: feedback G: gaming environmentS: social...
Experimental Design• 150 workers (30 per interface) were recruited  from Amazon Mechanical Turk within two  weeks.• Each w...
Gaming Environment Helps Collect       More High-Quality LabelsInterfaces withGaming Environmentcollect significantlymore ...
Social Interaction Motivates              More VolunteersInterfaces withSocial Interactionmotivate morevolunteers thanothe...
Implications for Human Computation1. Gaming Environment allows output   agreement games to collect more high-   quality ou...
Questions?
Upcoming SlideShare
Loading in …5
×

Systematic analysis of oag v4

1,705 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,705
On SlideShare
0
From Embeds
0
Number of Embeds
1,446
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Statistically significant
  • Systematic analysis of oag v4

    1. 1. Systematic Analysis ofOutput Agreement Games:Effects of Gaming Environment,Social Interaction, and FeedbackShih-Wen Huang and Wai-Tat Fu
    2. 2. GWAP Today – ESP Game [Von Ahn and Dabbish, CHI 2004]
    3. 3. GWAP Today - Foldit [Cooper et al., Nature 2010]
    4. 4. Two Unanswered Questions1. Can GWAP collect outputs with high quality? Why?2. Which component of GWAP motivate volunteers to work?• To answer these questions, a systematic analysis that tests the effects of different components of GWAP needs to be done
    5. 5. Output Agreement Games
    6. 6. Output Agreement Games Database1. Randomly matchmultiple strangersand provide them withthe same input Game Interface
    7. 7. Output Agreement Games Database1. Randomly matchmultiple strangersand provide them withthe same input Game Interface2. Players are asked togenerate outputs that arerelated to the input
    8. 8. Output Agreement Games Database1. Randomly match 3. Players will be rewardedmultiple strangers if the outputs are matched.and provide them with The matched outputs wouldthe same input be used to describe the input Game Interface2. Players are asked togenerate outputs that arerelated to the input
    9. 9. Three Major Components• Gaming Environment (G): Encouraging players to generate matched answers using scoring system, leaderboard, etc.• Social Interaction (S): Players need to cooperate with other players• Feedback (F): Players can implicitly learn the answers of other players
    10. 10. Five Different Interfaces• (B)• (F)• (F+S)• (F+G)• (F+S+G)B: baseline F: feedback G: gaming environmentS: social interaction
    11. 11. Five Different Interfaces• (B)• (F)• (F+S)• (F+G)• (F+S+G)B: baseline F: feedback G: gaming environmentS: social interaction
    12. 12. Five Different Interfaces• (B)• (F)• (F+S)• (F+G)• (F+S+G) Teammate searching page Teammate waiting pageB: baseline F: feedback G: gaming environmentS: social interaction
    13. 13. Five Different Interfaces• (B)• (F)• (F+S)• (F+G)• (F+S+G)B: baseline F: feedback G: gaming environmentS: social interaction
    14. 14. Five Different Interfaces• (B)• (F)• (F+S)• (F+G)• (F+S+G)B: baseline F: feedback G: gaming environmentS: social interaction
    15. 15. Experimental Design• 150 workers (30 per interface) were recruited from Amazon Mechanical Turk within two weeks.• Each worker earned $0.05 for generating 20 labels• A label was considered as a high-quality label if its distance to a gold standard label < 1
    16. 16. Gaming Environment Helps Collect More High-Quality LabelsInterfaces withGaming Environmentcollect significantlymore high qualitylabels than otherthree interfaces Amount of high-quality labels collected by each interface
    17. 17. Social Interaction Motivates More VolunteersInterfaces withSocial Interactionmotivate morevolunteers thanother threeinterfaces Percentage of workers reported that they would love to do the task again voluntarily
    18. 18. Implications for Human Computation1. Gaming Environment allows output agreement games to collect more high- quality outputs2. Social Interaction helps motivate more volunteers in output agreement games
    19. 19. Questions?

    ×