Your SlideShare is downloading. ×
Easy Mechanical Turk Integration with Houdini
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Easy Mechanical Turk Integration with Houdini

1,341

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,341
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Houdini On Demand Labor API Easy Mechanical Turk Integration with Houdini Philly AWS December Meetup Date: December 13, 2010 By: Chris Conley and Jameson Detweiler 12/13/2010Tuesday, December 14, 2010
  • 2. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Mechanical Turk • Marketplace for work • ~ 500k workers • Accessible via API 12/13/2010Tuesday, December 14, 2010
  • 3. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Task Examples • Moderation - images, comments, reviews, video • Categorization - products, best image, tagging • Ads - moderation, keywords • Audio Transcription / Video Captioning • Cleanup - proofreading, contact reconciliation, review editing • PDF/Image/Handwriting to Text • Research - contact info, document search 12/13/2010Tuesday, December 14, 2010
  • 4. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Turk Overview • Workers and Requesters • HITs • Questions • Assignments • Qualifications • Notifications • Blocking Workers 12/13/2010Tuesday, December 14, 2010
  • 5. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Worker View 12/13/2010Tuesday, December 14, 2010
  • 6. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Creating HITs WYSIWYG API • Easy • More Flexible • Great for testing • But Complicated the waters • Programmatic • No integration Integration 12/13/2010Tuesday, December 14, 2010
  • 7. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup WYSIWYG 12/13/2010Tuesday, December 14, 2010
  • 8. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup WYSIWYG 12/13/2010Tuesday, December 14, 2010
  • 9. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup API http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html 12/13/2010Tuesday, December 14, 2010
  • 10. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup HIT Questions • XML • External Question 12/13/2010Tuesday, December 14, 2010
  • 11. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Turk API Libraries • ruby-aws http://rubyforge.org/projects/ruby-aws/ • rTurk https://github.com/mdp/rturk • Turkee https://github.com/aantix/turkee 12/13/2010Tuesday, December 14, 2010
  • 12. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Questions about Turk? 12/13/2010Tuesday, December 14, 2010
  • 13. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Houdini • Simple • Qualified results • Automatic • Trusted Workers 12/13/2010Tuesday, December 14, 2010
  • 14. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Houdini APIs Generic Optimized • Simple • Image Moderation • Exact • Ad Moderation • Multi-answer • Data Reconciliation • Data Extraction • ... 12/13/2010Tuesday, December 14, 2010
  • 15. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Simple API • One answer per task • Cheap • Image Moderation, Low profile content 12/13/2010Tuesday, December 14, 2010
  • 16. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Simple API Example 12/13/2010Tuesday, December 14, 2010
  • 17. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Exact API • Multiple workers • Ensures all answers match • Returns the “final answer” • If not, escalated to trusted worker group • More expensive than Simple • Image Moderation, Data Harvesting, etc 12/13/2010Tuesday, December 14, 2010
  • 18. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Exact API Example 12/13/2010Tuesday, December 14, 2010
  • 19. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Multi-answer API • Ask multiple turkers same question • All answers are returned • Surveys, feedback, answer qualification on your end 12/13/2010Tuesday, December 14, 2010
  • 20. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Image Moderation API • No need to build form_html • Just send the image url • Optionally send custom instructions and possible responses 12/13/2010Tuesday, December 14, 2010
  • 21. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Image Moderation Example 12/13/2010Tuesday, December 14, 2010
  • 22. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Image Moderation Demo Source: https://github.com/chrisconley/cat_or_not 12/13/2010Tuesday, December 14, 2010
  • 23. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Admin Console • Review Work • Perform “Trusted” tasks • Worker Insights: Avg Completion Time, # of workers • Remove bad performing workers from specific task groups 12/13/2010Tuesday, December 14, 2010
  • 24. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Getting Started • Get an API Key - http://houdiniapi.com/sign-up/ • Tunnlr - http://tunnlr.com • Sandbox - http://houdini-sandbox.heroku.com 12/13/2010Tuesday, December 14, 2010
  • 25. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Pricing Example Monthly Fee Images Per Month $20 200 $99 1500 12/13/2010Tuesday, December 14, 2010
  • 26. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Other Solutions • Crowdflower • Microtask • CrowdCloud 12/13/2010Tuesday, December 14, 2010
  • 27. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Resources Houdini Docs: http://houdiniapi.com/documentation Mturk Docs: http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html 12/13/2010Tuesday, December 14, 2010
  • 28. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Thanks! Chris Conley Jameson Detweiler chris@houdiniapi.com jameson@houdiniapi.com @chrisconley @jaymstr 12/13/2010Tuesday, December 14, 2010

×