TurKit is a toolkit for building human computation algorithms. It allows researchers to integrate human input into algorithms through tasks on Amazon Mechanical Turk. The document describes two example systems (VizWiz and Soylent) that use TurKit and human computation to provide services: VizWiz helps blind users get descriptions of visual scenes, while Soylent integrates crowd workers directly into Microsoft Word for tasks like proofreading. TurKit provides tools for designing iterative and parallel human computation workflows.

1. 1TurKit: A Toolkit for Human Computation AlgorithmsRob Miller & Greg LittleUser Interface Design GroupMIT CSAILJoint work with Lydia Chilton, Max Goldman, Jeff Bigham, Aubrey Tatarowicz, Rajeev Najak, Michael BernsteinJune 10, 2010

2. OutlineIterative human computationTurKit: a toolkit for human computation algorithmsSystems with human computation insideVizWiz: vision for blind users with camera phonesSoylent: putting a crowd inside Microsoft WordJune 10, 20102

3. What is TurKit?

4. 2 WaysWhat is TurKit?

5. Way 1

6. Way 1ADD 1, 2

7. Way 1ADD 1, 2JMP somewhere

8. Way 1ADD 1, 2JMP somewhere…

9. Way 1ADD 1, 2JMP somewhere…TURK “How do you feel?”

10. Way 1ADD 1, 2JMP somewhere…TURK “How do you feel?”

11. Way 2

12. Way 2

13. Way 2

14. Way 2

15. Way 2

16. Way 2

17. Way 2

18. Way 2

19. Way 2

20. Way 2

21. Demo

22. DemoTurKit

23. What can you do with TurKit?

24. x50

25. 3119

28. Improve

29. Improve

30. VoteImprove

31. VoteImprove

32. Handwriting Recognition

33. Image Description

34. Outline to ProseProgression of Other Offer Description: $10,000 more

35. better package

36. far more competitive offer

37. more competitive offer

38. competing offerPicture Sorting

39. Subjective Ratingsx

40. Subjective Ratings

41. Subjective Ratings

42. Workflow Comparisons

43. Workflow Comparisons

44. Workflow Comparisons

45. Workflow Comparisons

46. Workflow Comparisons

47. Iterative

48. ParallelIterative

49. ParallelIterative

50. ParallelIterative

51. Which is better?

52. Which is better?30 images

53. Which is better?30 images6 iterations

54. Which is better?30 images6 iterations

55. Which is better?

56. Which is better?

57. Which is better?

58. Larger Algorithms

59. Larger Algorithms

60. Larger Algorithms

61. OutlineIterative human computationTurKit: a toolkit for human computation algorithmsSystems with human computation insideVizWiz: vision for blind users with camera phonesSoylent: putting a crowd inside Microsoft WordJune 10, 201058

62. VizWiz: Helping the Blind See59joint work with Jeff Bigham (University of Rochester)Which door is the women’s restroom?June 10, 2010

63. Helping the Blind See60MechanicalTurkthe left oneLEFTon the leftJune 10, 2010

64. Field StudyField deployment with 11 blind iPhone usersAnswers received within a minute for ~5 cents a questionLatency can be reduced to less than 30 seconds by keeping workers warmed up (at $4 per hour)June 10, 201061

65. All Mobile Users Are Situationally Disabled62joint work with Rajeev NajakMechanicalTurkWhen is Barack Obama speaking at MIT?June 10, 2010

66. Soylent: Putting a Crowd inside MS WordJune 10, 2010joint work with Michael Bernstein63

67. Crowd-Driven ProofreadingJune 10, 201064