This document describes a study that analyzed the effects of different components of games with a purpose (GWAP) on output quality and motivation. The study tested 5 interfaces that varied the inclusion of feedback, social interaction, and gaming environment components. It found that interfaces with gaming environment components collected significantly more high-quality labels. It also found that interfaces with social interaction motivated more volunteers to participate voluntarily in the future. The study provides empirical evidence that gaming environment improves output quality in GWAPs, while social interaction increases participant motivation.
4. Two Unanswered Questions
1. Can GWAP collect outputs with high quality?
Why?
2. Which component of GWAP motivate
volunteers to work?
• To answer these questions, a systematic analysis
that tests the effects of different components of
GWAP needs to be done
6. Output Agreement Games
Database
1. Randomly match
multiple strangers
and provide them with
the same input
Game Interface
7. Output Agreement Games
Database
1. Randomly match
multiple strangers
and provide them with
the same input
Game Interface
2. Players are asked to
generate outputs that are
related to the input
8. Output Agreement Games
Database
1. Randomly match 3. Players will be rewarded
multiple strangers if the outputs are matched.
and provide them with The matched outputs would
the same input be used to describe the input
Game Interface
2. Players are asked to
generate outputs that are
related to the input
9. Three Major Components
• Gaming Environment (G): Encouraging players
to generate matched answers using scoring
system, leaderboard, etc.
• Social Interaction (S): Players need to
cooperate with other players
• Feedback (F): Players can implicitly learn the
answers of other players
10. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction
11. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction
12. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G) Teammate searching page Teammate waiting page
B: baseline F: feedback G: gaming environment
S: social interaction
13. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction
14. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction
15. Experimental Design
• 150 workers (30 per interface) were recruited
from Amazon Mechanical Turk within two
weeks.
• Each worker earned $0.05 for generating 20
labels
• A label was considered as a high-quality label
if its distance to a gold standard label < 1
16. Gaming Environment Helps Collect
More High-Quality Labels
Interfaces with
Gaming Environment
collect significantly
more high quality
labels than other
three interfaces
Amount of high-quality labels collected by each interface
17. Social Interaction Motivates
More Volunteers
Interfaces with
Social Interaction
motivate more
volunteers than
other three
interfaces
Percentage of workers reported that they
would love to do the task again voluntarily
18. Implications for Human Computation
1. Gaming Environment allows output
agreement games to collect more high-
quality outputs
2. Social Interaction helps motivate more
volunteers in output agreement games