Easy Mechanical Turk Integration with Houdini
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,579
On Slideshare
1,579
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
5
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Houdini On Demand Labor API Easy Mechanical Turk Integration with Houdini Philly AWS December Meetup Date: December 13, 2010 By: Chris Conley and Jameson Detweiler 12/13/2010Tuesday, December 14, 2010
  • 2. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Mechanical Turk • Marketplace for work • ~ 500k workers • Accessible via API 12/13/2010Tuesday, December 14, 2010
  • 3. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Task Examples • Moderation - images, comments, reviews, video • Categorization - products, best image, tagging • Ads - moderation, keywords • Audio Transcription / Video Captioning • Cleanup - proofreading, contact reconciliation, review editing • PDF/Image/Handwriting to Text • Research - contact info, document search 12/13/2010Tuesday, December 14, 2010
  • 4. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Turk Overview • Workers and Requesters • HITs • Questions • Assignments • Qualifications • Notifications • Blocking Workers 12/13/2010Tuesday, December 14, 2010
  • 5. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Worker View 12/13/2010Tuesday, December 14, 2010
  • 6. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Creating HITs WYSIWYG API • Easy • More Flexible • Great for testing • But Complicated the waters • Programmatic • No integration Integration 12/13/2010Tuesday, December 14, 2010
  • 7. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup WYSIWYG 12/13/2010Tuesday, December 14, 2010
  • 8. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup WYSIWYG 12/13/2010Tuesday, December 14, 2010
  • 9. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup API http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html 12/13/2010Tuesday, December 14, 2010
  • 10. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup HIT Questions • XML • External Question 12/13/2010Tuesday, December 14, 2010
  • 11. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Turk API Libraries • ruby-aws http://rubyforge.org/projects/ruby-aws/ • rTurk https://github.com/mdp/rturk • Turkee https://github.com/aantix/turkee 12/13/2010Tuesday, December 14, 2010
  • 12. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Questions about Turk? 12/13/2010Tuesday, December 14, 2010
  • 13. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Houdini • Simple • Qualified results • Automatic • Trusted Workers 12/13/2010Tuesday, December 14, 2010
  • 14. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Houdini APIs Generic Optimized • Simple • Image Moderation • Exact • Ad Moderation • Multi-answer • Data Reconciliation • Data Extraction • ... 12/13/2010Tuesday, December 14, 2010
  • 15. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Simple API • One answer per task • Cheap • Image Moderation, Low profile content 12/13/2010Tuesday, December 14, 2010
  • 16. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Simple API Example 12/13/2010Tuesday, December 14, 2010
  • 17. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Exact API • Multiple workers • Ensures all answers match • Returns the “final answer” • If not, escalated to trusted worker group • More expensive than Simple • Image Moderation, Data Harvesting, etc 12/13/2010Tuesday, December 14, 2010
  • 18. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Exact API Example 12/13/2010Tuesday, December 14, 2010
  • 19. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Multi-answer API • Ask multiple turkers same question • All answers are returned • Surveys, feedback, answer qualification on your end 12/13/2010Tuesday, December 14, 2010
  • 20. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Image Moderation API • No need to build form_html • Just send the image url • Optionally send custom instructions and possible responses 12/13/2010Tuesday, December 14, 2010
  • 21. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Image Moderation Example 12/13/2010Tuesday, December 14, 2010
  • 22. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Image Moderation Demo Source: https://github.com/chrisconley/cat_or_not 12/13/2010Tuesday, December 14, 2010
  • 23. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Admin Console • Review Work • Perform “Trusted” tasks • Worker Insights: Avg Completion Time, # of workers • Remove bad performing workers from specific task groups 12/13/2010Tuesday, December 14, 2010
  • 24. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Getting Started • Get an API Key - http://houdiniapi.com/sign-up/ • Tunnlr - http://tunnlr.com • Sandbox - http://houdini-sandbox.heroku.com 12/13/2010Tuesday, December 14, 2010
  • 25. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Pricing Example Monthly Fee Images Per Month $20 200 $99 1500 12/13/2010Tuesday, December 14, 2010
  • 26. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Other Solutions • Crowdflower • Microtask • CrowdCloud 12/13/2010Tuesday, December 14, 2010
  • 27. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Resources Houdini Docs: http://houdiniapi.com/documentation Mturk Docs: http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html 12/13/2010Tuesday, December 14, 2010
  • 28. Houdini Easy Mechanical Turk Integration with Houdini On Demand Labor API Philly AWS December Meetup Thanks! Chris Conley Jameson Detweiler chris@houdiniapi.com jameson@houdiniapi.com @chrisconley @jaymstr 12/13/2010Tuesday, December 14, 2010