TurKit: Tools for Iterative Tasks on Mechanical Turk<br />Paper by Greg Little, Lydia B. Chilton, Rob Miller, and Max Gold...
Human Computation<br />There are still some tasks that are too difficult for computers to do well.<br />Examples:<br />Lab...
Motivation and Participation<br />Why would humans want to do these things?<br />Reputation (Q&A Sites, Review Sites)<br /...
Amazon Mechanical Turk<br />Marketplace for “Human Intelligence Tasks” (HITs).<br />Small amounts of money per task.<br />
Amazon Mechanical Turk<br />
Traditional Workflow<br />HIT<br />HIT<br />Data Collected in CSV File<br />Requester Posts HIT Group to Mechanical Turk<b...
Traditional Workflow: Pros & Cons<br />Easy to run simple, parallelized tasks.<br />Not so easy to run tasks in which turk...
Iterative Improvement and Voting<br />improved<br />text<br />output<br />original<br />Vote<br />Improve<br />
“Improve” and “Vote” Tasks<br />Improve-HIT: Improve a given body of text according to some instructions.<br />Vote-HIT: C...
“Improve Task”<br />Please improve the description for this image.<br />People will vote whether to approve your changes.<...
Image Description<br />Version 1:<br />A parial view of a pocket calculator together with some coins and a pen.<br />
Image Description<br />Version 2: <br />A view of personal items a calculator, and some gold and copper coins, and a round...
Image Description<br />Version 3:<br />A close-up photograph of the following items:<br />A CASIO multi-function calculato...
Image Description<br />Version 4:<br />[Builds on Version 3]<br />…Various British coins; two of £1 value, three of 20p va...
Experiment: Iterative vs. Single HIT<br />11 image description tasks<br />In each task:<br />Image selected randomly from ...
Experiment: Iterative vs. Single HIT<br />20 Turkers voted on each pair.<br />Votes favored the iteratively generated desc...
Writing Tasks: Outline to Prose<br />Improve letter to convey this outline:<br />Somehow I need this to sound diplomatic:<...
Writing Tasks: Outline to Prose<br />Improve letter to convey this outline:<br />Somehow I need this to sound diplomatic:<...
Writing Tasks: Outline to Prose<br />Improve letter to convey this outline:<br />Somehow I need this to sound diplomatic:<...
Handwriting Recognition<br />Version 1:<br />You (?) (?) (?) (work). (?) (?) (?) work (not) (time). I (?) (?) a few gramma...
Handwriting Recognition<br />Version 2:<br />You (?) (?) (?) (work). (?) (?) (?) work (not) (time). I (?) (?) a few gramma...
Handwriting Recognition<br />Version 4:<br />You (misspelled) (several) (words). (?) (?) (?) work next (time). I also noti...
Handwriting Recognition<br />Version 5:<br />You (misspelled) (several) (words). (Plan?) (spellcheck) (your) work next tim...
Handwriting Recognition<br />Final Version:<br />You (misspelled) (several) (words). Please spellcheck your work next time...
Task Time & Cost<br />
The TurKit Toolkit<br />Turkers<br />Arrows indicate the flow of information.<br />Programmer writes 2 sets of source code...
TurKit APIs<br />MTurk API: JavaScript wrapper for the MTurk API<br />Trace API: Uses the database to store information ab...
TurKit Demo<br />Iterative Text Improvement<br />
TurKit Demo: Properties<br />Fields to define:<br />Mode = {Sandbox, Offline, Real}<br />maxMoney & maxHITs = budget, HIT ...
TurKit Demo: Code<br />// improve text<br />varhitID = createImproveHIT(text, 0.02)<br />var hit = mturk.waitForHIT(hitId)...
TurKit Demo: Code<br />// verify improvement<br />if (vote(text, newText, 0.01)) {<br />text = newText<br />mturk.approveA...
TurKit Demo: Code<br />HITs are created using XML schemas defined by Amazon.<br />Referred to by URL:<br />Example:http://...
Upcoming SlideShare
Loading in …5
×

TurKit: Tools for Iterative Tasks on Mechanical Turk [Little, et al. 2010]

2,128 views
1,975 views

Published on

These are slides from a presentation I gave on the TurKit paper and toolkit by Greg Little and others at MIT.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,128
On SlideShare
0
From Embeds
0
Number of Embeds
180
Actions
Shares
0
Downloads
32
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Some trade-offs here. These are AI-Hard tasks, so we need humans to do it.This method may take a long amount of time, but is fairly reliable and cheap!
  • Programmer writes HTML\\XML files for HITsJavaScript for scripting program flow
  • TurKit: Tools for Iterative Tasks on Mechanical Turk [Little, et al. 2010]

    1. 1. TurKit: Tools for Iterative Tasks on Mechanical Turk<br />Paper by Greg Little, Lydia B. Chilton, Rob Miller, and Max Goldman (MIT CSAIL)<br />Presented by Sanjay Kairam (Stanford)<br />
    2. 2. Human Computation<br />There are still some tasks that are too difficult for computers to do well.<br />Examples:<br />Labeling Images<br />Tagging Documents<br />Proofreading Text<br />Writing Novel Content<br />Simple solution: Get humans to do it!<br />
    3. 3. Motivation and Participation<br />Why would humans want to do these things?<br />Reputation (Q&A Sites, Review Sites)<br />Contribution (Wikipedia)<br />Fun (Games with a Purpose)<br />$$$<br />
    4. 4. Amazon Mechanical Turk<br />Marketplace for “Human Intelligence Tasks” (HITs).<br />Small amounts of money per task.<br />
    5. 5. Amazon Mechanical Turk<br />
    6. 6. Traditional Workflow<br />HIT<br />HIT<br />Data Collected in CSV File<br />Requester Posts HIT Group to Mechanical Turk<br />HIT<br />Data Exported for Use<br />HIT<br />HIT<br />HIT<br />
    7. 7. Traditional Workflow: Pros & Cons<br />Easy to run simple, parallelized tasks.<br />Not so easy to run tasks in which turkers improve on or validate each others’ work.<br />TurKit to the rescue!<br />
    8. 8. Iterative Improvement and Voting<br />improved<br />text<br />output<br />original<br />Vote<br />Improve<br />
    9. 9. “Improve” and “Vote” Tasks<br />Improve-HIT: Improve a given body of text according to some instructions.<br />Vote-HIT: Choose which of two bodies of text is better according to some criteria.<br />
    10. 10. “Improve Task”<br />Please improve the description for this image.<br />People will vote whether to approve your changes.<br />Use no more than 500 characters.<br />“Vote Task”<br />Please select the better description for this image.<br />Your vote must agree with the majority to be approved.<br />Differences are highlighted in yellow.<br />Image Description<br />
    11. 11. Image Description<br />Version 1:<br />A parial view of a pocket calculator together with some coins and a pen.<br />
    12. 12. Image Description<br />Version 2: <br />A view of personal items a calculator, and some gold and copper coins, and a round tip pen, these are all pocket and wallet sized item used for business, writing, calculating prices or solving math problems and purchasing items.<br />[Voted down in favor of Version 1]<br />
    13. 13. Image Description<br />Version 3:<br />A close-up photograph of the following items:<br />A CASIO multi-function calculator<br />A ball point pen, uncapped<br />Various coins, apparently European, both copper and gold<br />Seems to be a theme illustration for a brochure or document cover treating finance, probably personal finance.<br />
    14. 14. Image Description<br />Version 4:<br />[Builds on Version 3]<br />…Various British coins; two of £1 value, three of 20p value and one of 1p value...<br />Further versions continued to iterate on Version 3.<br />
    15. 15. Experiment: Iterative vs. Single HIT<br />11 image description tasks<br />In each task:<br />Image selected randomly from set of 10<br />Budget chosen (either $0.25 or $0.50)<br />Iterative HIT Condition: $0.02/Improve & $0.01/Vote<br />Single HIT Condition: Single HIT with $0.25 or $0.50 Reward<br />
    16. 16. Experiment: Iterative vs. Single HIT<br />20 Turkers voted on each pair.<br />Votes favored the iteratively generated description in 9 out of 11 cases (82%).<br />
    17. 17. Writing Tasks: Outline to Prose<br />Improve letter to convey this outline:<br />Somehow I need this to sound diplomatic:<br />I am grateful for job offer<br />I got another offer<br />they pay $10,000 more<br />I’ll work for you if you can match their offer<br />Version 1:<br />I want to thank you for your gracious job offer. However, I have had another job offer willing to pay me $10,000 more for the same job. But, I am willing to forgoe that job offer because working with your company would be a great opportunity, that is, if you would like to match their offer.<br />
    18. 18. Writing Tasks: Outline to Prose<br />Improve letter to convey this outline:<br />Somehow I need this to sound diplomatic:<br />I am grateful for job offer<br />I got another offer<br />they pay $10,000 more<br />I’ll work for you if you can match their offer<br />Version 3:<br />I would like to thank you very much for your offer, however, I have received a better package from another firm. I still would like to work for company and would like to speak with you to see if there is anyway we could make that occur.<br />
    19. 19. Writing Tasks: Outline to Prose<br />Improve letter to convey this outline:<br />Somehow I need this to sound diplomatic:<br />I am grateful for job offer<br />I got another offer<br />they pay $10,000 more<br />I’ll work for you if you can match their offer<br />Version 9:<br />Thank you very much for your offer. At this time, I have a competing offer on the table at a higher salary. If the remuneration packages were equal I would much prefer to work for yourselves, and I am open to discussing the possibility of your matching this offer. If this is possible, please contact me at your earliest convenience.<br />
    20. 20. Handwriting Recognition<br />Version 1:<br />You (?) (?) (?) (work). (?) (?) (?) work (not) (time). I (?) (?) a few grammatical mistakes. Overall your writing style is a bit too (phoney). You do (?) have good (points), but they got lost amidst the (writing). (signature)<br />
    21. 21. Handwriting Recognition<br />Version 2:<br />You (?) (?) (?) (work). (?) (?) (?) work (not) (time). I (?) (?) a few grammatical mistakes. Overall your writing style is a bit too (phoney). You do (?) have good (points), but they got lost amidst the (writing). (signature)<br />
    22. 22. Handwriting Recognition<br />Version 4:<br />You (misspelled) (several) (words). (?) (?) (?) work next (time). I also notice a few grammatical mistakes. …<br />
    23. 23. Handwriting Recognition<br />Version 5:<br />You (misspelled) (several) (words). (Plan?) (spellcheck) (your) work next time. I also notice a few grammatical mistakes. Overall your writing style is a bit too phoney. You do make some good (points), but they got lost amidst the (writing). (signature)<br />
    24. 24. Handwriting Recognition<br />Final Version:<br />You (misspelled) (several) (words). Please spellcheck your work next time. I also notice a few grammatical mistakes. Overall your writing style is a bit too phoney. You do make some good (points), but they got lost amidst the (writing). (signature)<br />
    25. 25. Task Time & Cost<br />
    26. 26. The TurKit Toolkit<br />Turkers<br />Arrows indicate the flow of information.<br />Programmer writes 2 sets of source code:<br />HTML files for web servers<br />JavaScript executed by TurKit<br />Output is retrieved via a JavaScript database.<br />Mechanical Turk<br />Web Server<br />TurKit<br />JavaScript Database<br />*.js<br />*.html<br />Programmer<br />
    27. 27. TurKit APIs<br />MTurk API: JavaScript wrapper for the MTurk API<br />Trace API: Uses the database to store information about program execution<br />Utility API: Covers some common higher level MTurk tasks.<br />waitForHit: accepts a HIT ID and returns a JavaScript object containing answers.<br />vote: manages a HIT where turkers vote between two or more options.<br />sort: takes two parameters and a comprator<br />
    28. 28. TurKit Demo<br />Iterative Text Improvement<br />
    29. 29. TurKit Demo: Properties<br />Fields to define:<br />Mode = {Sandbox, Offline, Real}<br />maxMoney & maxHITs = budget, HIT limits<br />repeatInterval = wait time before re-running script<br />You will also need:<br />AWS Developer Access Key<br />AWS Secret Key<br />
    30. 30. TurKit Demo: Code<br />// improve text<br />varhitID = createImproveHIT(text, 0.02)<br />var hit = mturk.waitForHIT(hitId)<br />varnewText = hit.assignments[0].answer.newText<br />print(“-------------”)<br />print(newText)<br />print(“-------------”)<br />
    31. 31. TurKit Demo: Code<br />// verify improvement<br />if (vote(text, newText, 0.01)) {<br />text = newText<br />mturk.approveAssignment(hit.assignments[0])<br />print(“nvote = keepn”)<br />}<br />else {<br />Mturk.rejectAssignment(hit.assignments[0])<br />print(“nvote = rejectn”)<br />
    32. 32. TurKit Demo: Code<br />HITs are created using XML schemas defined by Amazon.<br />Referred to by URL:<br />Example:http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2005-10-01/QuestionForm.xsd;<br />

    ×