Who We AreJim Jones ( jim.jones1@gmail.com )Creator of the Turkee gem, a Rails form library that allows adeveloper to easi...
What is Mechanical TurkThe Amazon Mechanical Turk (MTurk) is one of the suites ofAmazon Web Services, a crowd-sourcing Int...
Mechanical Turk Demographics http://waxy.org/2008/11/the_faces_of_mechanical_turk/
More Demographics http://behind-the-enemy-lines.blogspot.com/2010/03/new- demographics-of-mechanical-turk.html United Stat...
The Mechanical Turk Interface Worker Interface https://www.mturk.com/mturk/findhits? match=false Requester Interface https...
Common Uses Tagging of images Collection of survey data Sentiment analysis
Creative Uses of Mechanical Turk   10,000 Sheep All drawn by Turkers   http://www.thesheepmarket.com/   Dr. Suessify the N...
Iterative Uses for Mechanical TurkExploring Iterative and Parallel Human ComputationProcesses Greg Little1, Lydia B. Chilt...
Ruby Integration RTurk gem http://github.com/mdp/rturk Creating HITs h = RTurk::Hit.create(:title => hit_title) do |hit|  ...
Rails Integration Turkee Gem   Turkee gem http://github.com/aantix/turkeeThe Turkee gem builds on top of RTurk providing a...
Retrieving Data from MechanicalTurkThe process_hits method retrieves the user posted data fromMechanical Turk, reconstruct...
Turkee Iterator (updated for Rails3.1)Turkee Iterator is a demo built on top of Turkee to sample some ofthe capabilities o...
Christmas Demos
Gift For WifeDirections : Should be under $25 Should include a link to theitem on Amazon Should look like I really care.Ei...
Difficult DilemmaDirections: I’m having difficulty telling my kids that Santa doesn’treally exist. On top of that, they’e ...
Let’s Do a Realtime Example, SayHellohttp://turkee31.herokuapp.com/turkee/turkee_tasks/new   Please say “Hello” to my coll...
More Content Creation
Mechanical Turk is an untappedmarket in terms of getting contentcreated for your application. Startthinking outside the co...
Example: Obesity Testimonials(Design of Questions)Upload a picture of yourself of when youfirst considered yourself “fat”....
Other Concepts
Micro BonusesThe concept where the user is awarded a varying amountdepending upon how much time they take to answer a ques...
GamblingAllow users to gamble their potential pay.E.g. Sentiment analysis.Show the user all of the recent headlines for a ...
Interesting Research Findings
Financial Incentives and thePerformance of CrowdsWinter Mason, Yahoo! ResearchFigure 2 reveals two main findings: first, t...
Toward Automatic Task Design: AProgress ReportEric Huang, School of Engineering and Applied SciencesHarvard University
Task Search in a HumanComputation Market   Lydia B. Chilton, University of Washington **We found strong evidence that Turk...
Ruby, Turkee and Mechanical Turk
Ruby, Turkee and Mechanical Turk
Ruby, Turkee and Mechanical Turk
Upcoming SlideShare
Loading in …5
×

Ruby, Turkee and Mechanical Turk

2,692 views
2,558 views

Published on

Jim is the the creator of the Turkee gem, a Rails form library that allows a developer to easily integrate with Mechanical Turk using the standard Rails form helpers. The gem also makes it easy to retrieve the data submitted to Mechanical Turk and map those values back to your existing models.

Mark is the creator of the RTurk gem (upon which Turkee is built).

Turkee - Blurring the lines between Rails and Mechanical Turk. https://github.com/aantix/turkee

RTurk - A simple wrapper and library for Amazon's Mechanical Turk. https://github.com/mdp/rturk

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,692
On SlideShare
0
From Embeds
0
Number of Embeds
476
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Ruby, Turkee and Mechanical Turk

  1. 1. Who We AreJim Jones ( jim.jones1@gmail.com )Creator of the Turkee gem, a Rails form library that allows adeveloper to easily integrate with Mechanical Turk using thestandard Rails form helpers. The gem also makes it easy toretrieve the data submitted to Mechanical Turk and map thosevalues back to your existing models.Mark Percival ( m@mdp.im )Creator of the RTurk gem, a simple wrapper and library forAmazon’s Mechanical Turk API.
  2. 2. What is Mechanical TurkThe Amazon Mechanical Turk (MTurk) is one of the suites ofAmazon Web Services, a crowd-sourcing Internet marketplace thatenables computer programmers (known as Requesters) to co-ordinate the use of human intelligence to perform tasks whichcomputers are unable to do. The people that perform these tasksare informally referred to as “Turkers”.
  3. 3. Mechanical Turk Demographics http://waxy.org/2008/11/the_faces_of_mechanical_turk/
  4. 4. More Demographics http://behind-the-enemy-lines.blogspot.com/2010/03/new- demographics-of-mechanical-turk.html United States: 46.80% India: 34.00% Miscellaneous: 19.20%
  5. 5. The Mechanical Turk Interface Worker Interface https://www.mturk.com/mturk/findhits? match=false Requester Interface https://requester.mturk.com/bulk/batches
  6. 6. Common Uses Tagging of images Collection of survey data Sentiment analysis
  7. 7. Creative Uses of Mechanical Turk 10,000 Sheep All drawn by Turkers http://www.thesheepmarket.com/ Dr. Suessify the News http://groups.csail.mit.edu/uid/deneme/?p=671 Russia to launch 520-day mock mission to Mars ! Oh, The Places Russia Will Go! Fisherman’s wife breaks the silence ! One Fish Two Fish Sick Fish Who Fish BP tries again to cap well; protests set to start ! BP tries to cap with a hat while protesters start to raise hell about the well “Take a Bow” http://nowtakeabow.com/This isn’t created with Mechanical Turk, but you can envision thepossibilities of utilizing Mechanical Turk workers for the creation ofa piecemeal video such as this.
  8. 8. Iterative Uses for Mechanical TurkExploring Iterative and Parallel Human ComputationProcesses Greg Little1, Lydia B. Chilton2, Max Goldman1, RobertC. Miller1We are interested in human computation processes whichcoordinate small contributions from many humans to achievelarger goals. For example, an algorithm might coordinate manyworkers to write a description for an image.
  9. 9. Ruby Integration RTurk gem http://github.com/mdp/rturk Creating HITs h = RTurk::Hit.create(:title => hit_title) do |hit| hit.assignments = num_assignments hit.description = hit_description hit.reward = reward hit.lifetime = duration.days.seconds.to_i hit.question(f_url, :frame_height => HIT_FRAMEHEIGHT) hit.qualifications.add :approval_rate, { :gt => 80 } end Reviewing HITs hits = RTurk::Hit.all_reviewable puts “#{hits.size} reviewable hits. n” unless hits.empty? puts “Reviewing all assignments” hits.each do |hit| hit.assignments.each do |assignment| puts assignment.answers[‘tags’] assignment.approve! if assignment.status == ‘Submitted’ end end end
  10. 10. Rails Integration Turkee Gem Turkee gem http://github.com/aantix/turkeeThe Turkee gem builds on top of RTurk providing a simple formhelper for posting/representing model data on Mechanical Turk’sexternal servers.The create_hit method takes a model,creates a Mechanical Turk hit pointing it atthe new resource URL for the modelpassed in. e.g. new_iteration_urltask = Turkee::TurkeeTask.create_hit(host, turkee_task[:hit_title], turkee_task[: turkee_task[:hit_num_assignments], turkee_taThe form helper directs the form actiontowards the external Turk url and expandsthe fields in the given block in the samemanner that Rails’s form_for helper does.<form accept-charset="UTF-8" action="https://workersandbox.mturk.com/mturk/extern <input type="hidden" id="assignmentId" name="assignmentId" value="2G2UC26DG67 <input type="hidden" id="hitId" name="hitId" value="29E6NG1FS3NYNMBGNC6ZVZL21 <input type="hidden" id="workerId" name="workerId" value="A3DWBLF5GT7ACT"/> <input type="hidden" id="turkSubmitTo" name="turkSubmitTo" value="https%3A%2F <input id="iteration_turkee_task_id" name="survey[turkee_task_id]" type="hidd <p><textarea cols="40" id="survey_value" name="survey[value]" rows="20"></tex <p><input name="commit" type="submit" value="Create" /></p></form>
  11. 11. Retrieving Data from MechanicalTurkThe process_hits method retrieves the user posted data fromMechanical Turk, reconstructs the data into a param hash (like aController), determines the model for which the data represents,and creates an entry for that model in the database.Turkee::TurkeeTask.process_hits(@turkee_task)rake turkee:get_all_results
  12. 12. Turkee Iterator (updated for Rails3.1)Turkee Iterator is a demo built on top of Turkee to sample some ofthe capabilities of Mechanical Turk. Allows for flexible HIT requestsfrom Mechanical Turk.https://github.com/aantix/turkee_iterator_rails31
  13. 13. Christmas Demos
  14. 14. Gift For WifeDirections : Should be under $25 Should include a link to theitem on Amazon Should look like I really care.Either vote for your favorite submission OR give me your own giftidea (and Amazon link) for a chance to win a bonus 50 cents.Thanks for the help!http://turkee31.herokuapp.com/iterations?id=9
  15. 15. Difficult DilemmaDirections: I’m having difficulty telling my kids that Santa doesn’treally exist. On top of that, they’e been terrible this past year.Please write a short letter to my kids from Santa explaining why hecan no longer give them gifts anymore(any reason will do). I willtranscribe the best one it and leave it in their stockings onChristmas morning.Merry Christmas!http://turkee31.herokuapp.com/iterations?id=11
  16. 16. Let’s Do a Realtime Example, SayHellohttp://turkee31.herokuapp.com/turkee/turkee_tasks/new Please say “Hello” to my colleagues and tell us where you’re from. I am demo’ing Mechanical Turk to the Ruby SF meetup group. Please tell send us a creative “hello” and where you’re from. A one or two line introduction would be nice. Tell us how the weather is in your city. Tell us about your family. Or just simply say hello. 0.05 100 5 It’s mostly sunny at 65 degrees. Hello from San Mateo, CA!
  17. 17. More Content Creation
  18. 18. Mechanical Turk is an untappedmarket in terms of getting contentcreated for your application. Startthinking outside the context ofscientific surveys and fact findingand start thinking of this resourceas a pool of human workers withemotions, opinions, and stories totell.E.g. Iterative story telling where each Turker adds a piece to a never ending story. Have the Turker give their opinion on a topic or product and then have other Turkers respond to that opinion. Have the Turker describe some emotionally happy or painful point in their life. Ask emotionally charged question to have the Turker reveal the essence of who they are.
  19. 19. Example: Obesity Testimonials(Design of Questions)Upload a picture of yourself of when youfirst considered yourself “fat”.When the above picture was taken, were you aware thatyou had been gaining weight? Describe the feeling you hadwhen you first glanced at the picture and noticed yourweight gain. Did anyone else notice or make snidecomments? How did you feel?“As a child I was pretty average but I was lead to believe that Iwas fat. I would request simple items, things that are statussymbols to the average kid, like jeans. These small requests wouldbe shot down, my mother telling me that I was too big to fit into apair of jeans. I remember her telling me that when I was eight. Itstarted a cycle in my life where I felt that I was too large, too fat,for the things other children took for granted.”
  20. 20. Other Concepts
  21. 21. Micro BonusesThe concept where the user is awarded a varying amountdepending upon how much time they take to answer a question orhow accurate it (relative to the other user answers)E.g. Maybe there’s a trivia game where you record how long ittakes each user to answer a question and the fastest response forthe day gets 10 cents instead of 5.
  22. 22. GamblingAllow users to gamble their potential pay.E.g. Sentiment analysis.Show the user all of the recent headlines for a given stock, stockcharts, etc.You state that they can either take their payout now, OR, if theythink that the stock will close out higher than it is now at the endof the day, they can “gamble”, and will be awarded the fee PLUS abonus at the end of the day. If it closes down from the point atwhere it is, they receive nothing.Does a future, larger payout increase the quality of data analysis?
  23. 23. Interesting Research Findings
  24. 24. Financial Incentives and thePerformance of CrowdsWinter Mason, Yahoo! ResearchFigure 2 reveals two main findings: first, that across all difficultylevels participants chose to complete more tasks on average whenthe pay was higher (F(3,607) = 15.73, p < 0.001); and second,that across all payment levels, the number of completed tasksdecreased with increasing difficulty.As Figure 3 indicates, however, increasing compensation didnot improve accuracy, which we measured in two ways…
  25. 25. Toward Automatic Task Design: AProgress ReportEric Huang, School of Engineering and Applied SciencesHarvard University
  26. 26. Task Search in a HumanComputation Market Lydia B. Chilton, University of Washington **We found strong evidence that Turkers sort by the most number ofHITs available (so they can find one task, and then do 100instances of them in a row) and the most recently posted HITS (sothey get the latest and greatest HITs).

×