Gamification of crowdsourcing tasks: What motivates a medical expert?

1. Gamification of crowdsourcing tasks: What motivates a medical expert? 18 July 2014 Rens van Honschooten Lora Aroyo (VU) & Robert-Jan Sips (IBM)

2. What motivates medical experts?

3. Gamification of crowdsourcing tasks: What motivates a medical expert? Why incentives for a medical expert? ● Train Watson in the medical domain ● To train cognitive systems you need data ○ Lots of it ● To get annotated data:

4. Gamification of crowdsourcing tasks: What motivates a medical expert? Crowdsourcing games to collect data ● Large amounts of data, fast and cheap [1]

5. Gamification of crowdsourcing tasks: What motivates a medical expert? Crowdsourcing games to collect data ● One thing they all have in common? ● They motivate their crowd ● The challenge: find out how to motivate busy medical experts

6. Gamification of crowdsourcing tasks: What motivates a medical expert? Dr. Detective: finding clues for a diagnosis ● Learning features

7. Gamification of crowdsourcing tasks: What motivates a medical expert? Dr. Detective: finding clues for a diagnosis ● Learning features● Most experts liked playing it ● Downsides: ○ The scoring was not clear ○ No rewards for completing a task ● Incentives ○ Competition ○ Entertainment ○ Learning ● Are these all the incentives? ● Which are the most important?

8. Gamification of crowdsourcing tasks: What motivates a medical expert? Crowdsourcing incentives (literature) Moon, H., & Baek, Y. (2009). Exploring variables affecting player’s intrinsic motivation in educational games. In Proceedings of the 17th International Conference on Computers in Education (pp. 718-722). Fang, Y., & Neufeld, D. (2009). Understanding sustained participation in open source software projects. Journal of Management Information Systems, 25(4), 9-50. Incentive Example Personal interest Interested in science, medicine, etc. Contributing to the community Helping diabetes patients to eat healthy Status & recognition Being recognized as a good doctor Personal growth Learning new thing in their medical domain Enjoyment Solving challenging medical problems Choice autonomy Choosing what to do and when, at work A monetary reward Receiving money for helping patients

9. Gamification of crowdsourcing tasks: What motivates a medical expert?

10. Gamification of crowdsourcing tasks: What motivates a medical expert? Helping patients Improve medical field

11. Gamification of crowdsourcing tasks: What motivates a medical expert? ● Can we incorporate these incentives to motivate the experts? ● Created a design for next version of Dr. Detective, called Dr. Watson ○ evaluated mockup by interviewing and testing with 6 experts

12. Gamification of crowdsourcing tasks: What motivates a medical expert? The design of Dr. Watson

13. Gamification of crowdsourcing tasks: What motivates a medical expert? The design of Dr. Watson ● Personal growth, learning ● Reading most preferred ● Provide overview of literature ● Spark their interest

14. Gamification of crowdsourcing tasks: What motivates a medical expert? The design of Dr. Watson ● To unlock articles: ○ play games to gather reputation points ● Articles motivate medical experts ○ Except researchers and students ○ Medical books, videos, giveaways and accreditation points should motivate these experts

16. Gamification of crowdsourcing tasks: What motivates a medical expert? ● Select game mode ● Cannot explore all content ● Make the experts curious ● More difficult tasks, to keep the game challenging

18. Gamification of crowdsourcing tasks: What motivates a medical expert? ● Make game more challenging ● Motivate the experts to keep playing ○ motivate them to play less popular tasks and annotate articles without annotations

20. Gamification of crowdsourcing tasks: What motivates a medical expert? ● Goal: beat the score of the crowd ○ average amount of points earned by others ○ matching the score also means you win ● To get points ○ Agreement with other experts ○ Pioneer by suggesting new answer ○ Three or five in a row bonus ● The scoring was clear for the experts ○ streaks motivate them to annotate correctly ● To increase game difficulty use topics the expert is less familiar with

23. Gamification of crowdsourcing tasks: What motivates a medical expert? ● To support learning goals more topic selection is required ● Medical experts stated to play again ○ To learn more ○ To improve their score ○ Because they enjoyed playing it ● The experts could easily navigate through the UI ○ elements at the bottom of the sidebar ● Experts wanted to see more clearly how Watson is improving

24. Gamification of crowdsourcing tasks: What motivates a medical expert? Future work - improving Dr. Watson ● Topic selection for tasks ● Personalization of: ○ The difficulty - take specialization in account ○ The homepage - change background image, reorder elements ● Incorporate additional unlockables/rewards ○ Videos ○ Medical books ○ Accreditation points ● Add ‘Watson status’ page ○ track the progress of the crowd and what watson achieved

25. Gamification of crowdsourcing tasks: What motivates a medical expert? What motivates medical experts? Helping patients Improve medical field

26. Questions?