Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

TurKit: A Toolkit for Human Computation Algorithms

2,062 views

Published on

Published in: Technology
  • Be the first to comment

TurKit: A Toolkit for Human Computation Algorithms

  1. 1. 1<br />TurKit: A Toolkit for Human Computation Algorithms<br />Rob Miller & Greg Little<br />User Interface Design Group<br />MIT CSAIL<br />Joint work with Lydia Chilton, Max Goldman, Jeff Bigham, Aubrey Tatarowicz, Rajeev Najak, Michael Bernstein<br />June 10, 2010<br />
  2. 2. Outline<br />Iterative human computation<br />TurKit: a toolkit for human computation algorithms<br />Systems with human computation inside<br />VizWiz: vision for blind users with camera phones<br />Soylent: putting a crowd inside Microsoft Word<br />June 10, 2010<br />2<br />
  3. 3. What is TurKit?<br />
  4. 4. 2 Ways<br />What is TurKit?<br />
  5. 5. Way 1<br />
  6. 6. Way 1<br />ADD 1, 2<br />
  7. 7. Way 1<br />ADD 1, 2<br />JMP somewhere<br />
  8. 8. Way 1<br />ADD 1, 2<br />JMP somewhere<br />…<br />
  9. 9. Way 1<br />ADD 1, 2<br />JMP somewhere<br />…<br />TURK “How do you feel?”<br />
  10. 10. Way 1<br />ADD 1, 2<br />JMP somewhere<br />…<br />TURK “How do you feel?”<br />
  11. 11. Way 2<br />
  12. 12. Way 2<br />
  13. 13. Way 2<br />
  14. 14. Way 2<br />
  15. 15. Way 2<br />
  16. 16. Way 2<br />
  17. 17. Way 2<br />
  18. 18. Way 2<br />
  19. 19. Way 2<br />
  20. 20. Way 2<br />
  21. 21. Demo<br />
  22. 22. Demo<br />TurKit<br />
  23. 23. What can you do with TurKit?<br />
  24. 24. x50<br />
  25. 25. 31<br />19<br />
  26. 26.
  27. 27.
  28. 28. Improve<br />
  29. 29. Improve<br />
  30. 30. Vote<br />Improve<br />
  31. 31. Vote<br />Improve<br />
  32. 32. Handwriting Recognition<br />
  33. 33. Image Description<br />
  34. 34. Outline to Prose<br />Progression of Other Offer Description:<br /><ul><li> $10,000 more
  35. 35. better package
  36. 36. far more competitive offer
  37. 37. more competitive offer
  38. 38. competing offer</li></li></ul><li>Picture Sorting<br />
  39. 39. Subjective Ratings<br />x<br />
  40. 40. Subjective Ratings<br />
  41. 41. Subjective Ratings<br />
  42. 42. Workflow Comparisons<br />
  43. 43. Workflow Comparisons<br />
  44. 44. Workflow Comparisons<br />
  45. 45. Workflow Comparisons<br />
  46. 46. Workflow Comparisons<br />
  47. 47. Iterative<br />
  48. 48. Parallel<br />Iterative<br />
  49. 49. Parallel<br />Iterative<br />
  50. 50. Parallel<br />Iterative<br />
  51. 51. Which is better?<br />
  52. 52. Which is better?<br />30 images<br />
  53. 53. Which is better?<br />30 images<br />6 iterations<br />
  54. 54. Which is better?<br />30 images<br />6 iterations<br />
  55. 55. Which is better?<br />
  56. 56. Which is better?<br />
  57. 57. Which is better?<br />
  58. 58. Larger Algorithms<br />
  59. 59. Larger Algorithms<br />
  60. 60. Larger Algorithms<br />
  61. 61. Outline<br />Iterative human computation<br />TurKit: a toolkit for human computation algorithms<br />Systems with human computation inside<br />VizWiz: vision for blind users with camera phones<br />Soylent: putting a crowd inside Microsoft Word<br />June 10, 2010<br />58<br />
  62. 62. VizWiz: Helping the Blind See<br />59<br />joint work with Jeff Bigham (University of Rochester)<br />Which door is the women’s restroom?<br />June 10, 2010<br />
  63. 63. Helping the Blind See<br />60<br />MechanicalTurk<br />the left one<br />LEFT<br />on the left<br />June 10, 2010<br />
  64. 64. Field Study<br />Field deployment with 11 blind iPhone users<br />Answers received within a minute for ~5 cents a question<br />Latency can be reduced to less than 30 seconds by keeping workers warmed up (at $4 per hour)<br />June 10, 2010<br />61<br />
  65. 65. All Mobile Users Are Situationally Disabled<br />62<br />joint work with Rajeev Najak<br />MechanicalTurk<br />When is Barack Obama speaking at MIT?<br />June 10, 2010<br />
  66. 66. Soylent: Putting a Crowd inside MS Word<br />June 10, 2010<br />joint work with Michael Bernstein<br />63<br />
  67. 67. Crowd-Driven Proofreading<br />June 10, 2010<br />64<br />
  68. 68. Crowd-Driven Shortening<br />June 10, 2010<br />65<br />
  69. 69. Bigger Idea: REAL Wizard of Oz <br />Wizard of Oz is a tried-and-true prototyping technique in AI and HCI<br />Putting a human behind the curtain until we figure out how to put software there<br />Crowd computing enables Wizard of Oz systems that are useful and deployable<br />So we can start collecting data about how the system is really used in practice<br />Adding the AI backend becomes a performance or cost optimization<br />66<br />June 10, 2010<br />
  70. 70. Outline<br />Iterative human computation<br />TurKit: a toolkit for human computation algorithms<br />Systems with human computation inside<br />VizWiz: vision for blind users with camera phones<br />Soylent: putting a crowd inside Microsoft Word<br />Funded in part by <br />June 10, 2010<br />67<br />

×