SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
Gamification of crowdsourcing tasks: What motivates a medical expert?
3.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Why incentives for a medical expert?
● Train Watson in the medical domain
● To train cognitive systems you need data
○ Lots of it
● To get annotated data:
4.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Crowdsourcing games to collect data
● Large amounts of data, fast and cheap [1]
5.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Crowdsourcing games to collect data
● One thing they all have in common?
● They motivate their crowd
● The challenge: find out how to motivate busy medical
experts
6.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Dr. Detective: finding clues for a diagnosis
● Learning features
7.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Dr. Detective: finding clues for a diagnosis
● Learning features● Most experts liked playing it
● Downsides:
○ The scoring was not clear
○ No rewards for completing a task
● Incentives
○ Competition
○ Entertainment
○ Learning
● Are these all the incentives?
● Which are the most important?
8.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Crowdsourcing incentives
(literature)
Moon, H., & Baek, Y. (2009). Exploring variables affecting player’s intrinsic motivation in educational
games. In Proceedings of the 17th International Conference on Computers in Education (pp. 718-722).
Fang, Y., & Neufeld, D. (2009). Understanding sustained participation in open source software projects.
Journal of Management Information Systems, 25(4), 9-50.
Incentive Example
Personal interest Interested in science, medicine, etc.
Contributing to the community Helping diabetes patients to eat healthy
Status & recognition Being recognized as a good doctor
Personal growth Learning new thing in their medical domain
Enjoyment Solving challenging medical problems
Choice autonomy Choosing what to do and when, at work
A monetary reward Receiving money for helping patients
9.
Gamification of crowdsourcing tasks: What motivates a medical expert?
10.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Helping patients
Improve medical field
11.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Can we incorporate these incentives to motivate the
experts?
● Created a design for next version of Dr. Detective,
called Dr. Watson
○ evaluated mockup by interviewing and testing with 6 experts
12.
Gamification of crowdsourcing tasks: What motivates a medical expert?
The design of Dr. Watson
13.
Gamification of crowdsourcing tasks: What motivates a medical expert?
The design of Dr. Watson
● Personal growth,
learning
● Reading most
preferred
● Provide overview
of literature
● Spark their interest
14.
Gamification of crowdsourcing tasks: What motivates a medical expert?
The design of Dr. Watson
● To unlock articles:
○ play games to gather reputation points
● Articles motivate medical experts
○ Except researchers and students
○ Medical books, videos, giveaways and
accreditation points should motivate these
experts
15.
Gamification of crowdsourcing tasks: What motivates a medical expert?
16.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Select game mode
● Cannot explore all
content
● Make the experts
curious
● More difficult
tasks, to keep the
game challenging
17.
Gamification of crowdsourcing tasks: What motivates a medical expert?
18.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Make game more challenging
● Motivate the experts to keep playing
○ motivate them to play less popular tasks and
annotate articles without annotations
19.
Gamification of crowdsourcing tasks: What motivates a medical expert?
20.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Goal: beat the score of the crowd
○ average amount of points earned by others
○ matching the score also means you win
● To get points
○ Agreement with other experts
○ Pioneer by suggesting new answer
○ Three or five in a row bonus
● The scoring was clear for the experts
○ streaks motivate them to annotate correctly
● To increase game difficulty use topics
the expert is less familiar with
21.
Gamification of crowdsourcing tasks: What motivates a medical expert?
22.
Gamification of crowdsourcing tasks: What motivates a medical expert?
23.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● To support learning goals more topic selection is
required
● Medical experts stated to play again
○ To learn more
○ To improve their score
○ Because they enjoyed playing it
● The experts could easily navigate through the UI
○ elements at the bottom of the sidebar
● Experts wanted to see more clearly how Watson is
improving
24.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Future work - improving Dr. Watson
● Topic selection for tasks
● Personalization of:
○ The difficulty - take specialization in account
○ The homepage - change background image, reorder elements
● Incorporate additional unlockables/rewards
○ Videos
○ Medical books
○ Accreditation points
● Add ‘Watson status’ page
○ track the progress of the crowd and what watson achieved
25.
Gamification of crowdsourcing tasks: What motivates a medical expert?
What motivates medical experts?
Helping patients
Improve medical field