1.
Gamification of crowdsourcing tasks:
What motivates a medical expert?
18 July 2014
Rens van Honschooten
Lora Aroyo (VU) & Robert-Jan Sips (IBM)

2.
What motivates medical experts?

3.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Why incentives for a medical expert?
● Train Watson in the medical domain
● To train cognitive systems you need data
○ Lots of it
● To get annotated data:

4.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Crowdsourcing games to collect data
● Large amounts of data, fast and cheap [1]

5.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Crowdsourcing games to collect data
● One thing they all have in common?
● They motivate their crowd
● The challenge: find out how to motivate busy medical
experts

6.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Dr. Detective: finding clues for a diagnosis
● Learning features

7.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Dr. Detective: finding clues for a diagnosis
● Learning features● Most experts liked playing it
● Downsides:
○ The scoring was not clear
○ No rewards for completing a task
● Incentives
○ Competition
○ Entertainment
○ Learning
● Are these all the incentives?
● Which are the most important?

8.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Crowdsourcing incentives
(literature)
Moon, H., & Baek, Y. (2009). Exploring variables affecting player’s intrinsic motivation in educational
games. In Proceedings of the 17th International Conference on Computers in Education (pp. 718-722).
Fang, Y., & Neufeld, D. (2009). Understanding sustained participation in open source software projects.
Journal of Management Information Systems, 25(4), 9-50.
Incentive Example
Personal interest Interested in science, medicine, etc.
Contributing to the community Helping diabetes patients to eat healthy
Status & recognition Being recognized as a good doctor
Personal growth Learning new thing in their medical domain
Enjoyment Solving challenging medical problems
Choice autonomy Choosing what to do and when, at work
A monetary reward Receiving money for helping patients

9.
Gamification of crowdsourcing tasks: What motivates a medical expert?

10.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Helping patients
Improve medical field

11.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Can we incorporate these incentives to motivate the
experts?
● Created a design for next version of Dr. Detective,
called Dr. Watson
○ evaluated mockup by interviewing and testing with 6 experts

12.
Gamification of crowdsourcing tasks: What motivates a medical expert?
The design of Dr. Watson

13.
Gamification of crowdsourcing tasks: What motivates a medical expert?
The design of Dr. Watson
● Personal growth,
learning
● Reading most
preferred
● Provide overview
of literature
● Spark their interest

14.
Gamification of crowdsourcing tasks: What motivates a medical expert?
The design of Dr. Watson
● To unlock articles:
○ play games to gather reputation points
● Articles motivate medical experts
○ Except researchers and students
○ Medical books, videos, giveaways and
accreditation points should motivate these
experts

15.
Gamification of crowdsourcing tasks: What motivates a medical expert?

16.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Select game mode
● Cannot explore all
content
● Make the experts
curious
● More difficult
tasks, to keep the
game challenging

17.
Gamification of crowdsourcing tasks: What motivates a medical expert?

18.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Make game more challenging
● Motivate the experts to keep playing
○ motivate them to play less popular tasks and
annotate articles without annotations

19.
Gamification of crowdsourcing tasks: What motivates a medical expert?

20.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Goal: beat the score of the crowd
○ average amount of points earned by others
○ matching the score also means you win
● To get points
○ Agreement with other experts
○ Pioneer by suggesting new answer
○ Three or five in a row bonus
● The scoring was clear for the experts
○ streaks motivate them to annotate correctly
● To increase game difficulty use topics
the expert is less familiar with

21.
Gamification of crowdsourcing tasks: What motivates a medical expert?

22.
Gamification of crowdsourcing tasks: What motivates a medical expert?

23.
Gamification of crowdsourcing tasks: What motivates a medical expert?
● To support learning goals more topic selection is
required
● Medical experts stated to play again
○ To learn more
○ To improve their score
○ Because they enjoyed playing it
● The experts could easily navigate through the UI
○ elements at the bottom of the sidebar
● Experts wanted to see more clearly how Watson is
improving

24.
Gamification of crowdsourcing tasks: What motivates a medical expert?
Future work - improving Dr. Watson
● Topic selection for tasks
● Personalization of:
○ The difficulty - take specialization in account
○ The homepage - change background image, reorder elements
● Incorporate additional unlockables/rewards
○ Videos
○ Medical books
○ Accreditation points
● Add ‘Watson status’ page
○ track the progress of the crowd and what watson achieved

25.
Gamification of crowdsourcing tasks: What motivates a medical expert?
What motivates medical experts?
Helping patients
Improve medical field

26.
Questions?