Easy Mechanical Turk Integration with Houdini

1,806 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,806
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Easy Mechanical Turk Integration with Houdini

  1. 1. Houdini On Demand Labor API Easy Mechanical Turk Integration with Houdini Philly AWS December Meetup Date: December 13, 2010 By: Chris Conley and Jameson Detweiler 12/13/2010Tuesday, December 14, 2010
  2. 2. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Mechanical Turk • Marketplace for work • ~ 500k workers • Accessible via API 12/13/2010Tuesday, December 14, 2010
  3. 3. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Task Examples • Moderation - images, comments, reviews, video • Categorization - products, best image, tagging • Ads - moderation, keywords • Audio Transcription / Video Captioning • Cleanup - proofreading, contact reconciliation, review editing • PDF/Image/Handwriting to Text • Research - contact info, document search 12/13/2010Tuesday, December 14, 2010
  4. 4. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Turk Overview • Workers and Requesters • HITs • Questions • Assignments • Qualifications • Notifications • Blocking Workers 12/13/2010Tuesday, December 14, 2010
  5. 5. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Worker View 12/13/2010Tuesday, December 14, 2010
  6. 6. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Creating HITs WYSIWYG API • Easy • More Flexible • Great for testing • But Complicated the waters • Programmatic • No integration Integration 12/13/2010Tuesday, December 14, 2010
  7. 7. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup WYSIWYG 12/13/2010Tuesday, December 14, 2010
  8. 8. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup WYSIWYG 12/13/2010Tuesday, December 14, 2010
  9. 9. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup API http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html 12/13/2010Tuesday, December 14, 2010
  10. 10. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup HIT Questions • XML • External Question 12/13/2010Tuesday, December 14, 2010
  11. 11. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Turk API Libraries • ruby-aws http://rubyforge.org/projects/ruby-aws/ • rTurk https://github.com/mdp/rturk • Turkee https://github.com/aantix/turkee 12/13/2010Tuesday, December 14, 2010
  12. 12. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Questions about Turk? 12/13/2010Tuesday, December 14, 2010
  13. 13. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Houdini • Simple • Qualified results • Automatic • Trusted Workers 12/13/2010Tuesday, December 14, 2010
  14. 14. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Houdini APIs Generic Optimized • Simple • Image Moderation • Exact • Ad Moderation • Multi-answer • Data Reconciliation • Data Extraction • ... 12/13/2010Tuesday, December 14, 2010
  15. 15. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Simple API • One answer per task • Cheap • Image Moderation, Low profile content 12/13/2010Tuesday, December 14, 2010
  16. 16. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Simple API Example 12/13/2010Tuesday, December 14, 2010
  17. 17. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Exact API • Multiple workers • Ensures all answers match • Returns the “final answer” • If not, escalated to trusted worker group • More expensive than Simple • Image Moderation, Data Harvesting, etc 12/13/2010Tuesday, December 14, 2010
  18. 18. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Exact API Example 12/13/2010Tuesday, December 14, 2010
  19. 19. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Multi-answer API • Ask multiple turkers same question • All answers are returned • Surveys, feedback, answer qualification on your end 12/13/2010Tuesday, December 14, 2010
  20. 20. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Image Moderation API • No need to build form_html • Just send the image url • Optionally send custom instructions and possible responses 12/13/2010Tuesday, December 14, 2010
  21. 21. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Image Moderation Example 12/13/2010Tuesday, December 14, 2010
  22. 22. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Image Moderation Demo Source: https://github.com/chrisconley/cat_or_not 12/13/2010Tuesday, December 14, 2010
  23. 23. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Admin Console • Review Work • Perform “Trusted” tasks • Worker Insights: Avg Completion Time, # of workers • Remove bad performing workers from specific task groups 12/13/2010Tuesday, December 14, 2010
  24. 24. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Getting Started • Get an API Key - http://houdiniapi.com/sign-up/ • Tunnlr - http://tunnlr.com • Sandbox - http://houdini-sandbox.heroku.com 12/13/2010Tuesday, December 14, 2010
  25. 25. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Pricing Example Monthly Fee Images Per Month $20 200 $99 1500 12/13/2010Tuesday, December 14, 2010
  26. 26. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Other Solutions • Crowdflower • Microtask • CrowdCloud 12/13/2010Tuesday, December 14, 2010
  27. 27. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Resources Houdini Docs: http://houdiniapi.com/documentation Mturk Docs: http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html 12/13/2010Tuesday, December 14, 2010
  28. 28. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Thanks! Chris Conley Jameson Detweiler chris@houdiniapi.com jameson@houdiniapi.com @chrisconley @jaymstr 12/13/2010Tuesday, December 14, 2010

×