Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS Mechanical Turk Office Hours - Jan 2011

3,008 views

Published on

Slides presented at the Mechanical Turk Office Hours on January 20, 2011.

Video of the presentation can be found on the Mechanical Turk blog:
http://mechanicalturk.typepad.com

Published in: Technology
  • You can try to use this service ⇒ www.WritePaper.info ⇐ I have used it several times in college and was absolutely satisfied with the result.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

AWS Mechanical Turk Office Hours - Jan 2011

  1. 1. OFFICE HOURS<br />January 20, 2011<br />Presented by:<br />James Willeford, Principal Engineer<br />Adam Bradley, Sr. Engineer <br />Natala Menezes, Sr. Product Manager<br />
  2. 2. Welcome<br />We are here to participate in a conversation.<br />We want to hear your feedback &questions.<br />Ask a question with the questionpanel. <br />We will moderate questions. <br />Agenda<br />Introductions<br />Platform Review <br />Product & API Updates<br />Best Practices: HIT Design<br />Q & A<br />
  3. 3. Introductions<br />
  4. 4. How customers are using mTurk:<br />Categorization<br />Classification<br />Tagging <br />Sentiment Analysis<br />Data Management<br />Business Listing Verification by Phone or Web<br />Data Entry <br />Training Data (Algo) <br />Business Services<br />Product Testing<br />Surveys<br />Web Research <br />Search Relevancy<br />Content & Media<br />Photo & Content Moderation<br />Content Creation<br />Translation<br />Transcription <br />
  5. 5. How Requesters access mTurk:<br />
  6. 6. Applications & Solution Providers<br />Mechanical Turk supports a robust ecosystem. <br />
  7. 7. Product Updates<br />Worker Management <br />Released in October 2010<br />Search by Requester Name<br />Released in December 2010<br />Improved: In-progress Batch Approvals and Rejections<br />Feature extension that brings this functionality to upload/download via CSV in the Website UI<br />Released in January 2011<br />API Updates<br />
  8. 8. Worker Management<br />Makes it easier for you to find Workers who are good at your HITs.<br />Key Features:<br />Workers StatisticsData on who is doing work for you, their performance on that work.<br />Manage Qualification TypesCreate new Qualification Types in the UI and assign them to Workers to create Qualified Worker groups. <br />Worker Management ToolsBonus, Block and assign qualifications for a specific Worker.<br />
  9. 9. Workers Tab <br />See a list of all Workers who have submitted Work to you, their Qualification Types and Block status. Make modifications via CSV. <br />Customize your view by selecting the Qualification Types to be displayed (select up to 5). <br />New Tab! Found under the “Manage” tab. <br />Download a CSV of all your Workers. <br />You can assign a Qualification Type, change a Workers Qualification Score and modify their Block Status in the CSV and then Upload the CSV back to Mechanical Turk to process the changes. <br />Click on a Worker ID to take action on an individual Worker.<br />
  10. 10. Individual Worker Details<br />Access any worker with their ID at this URL: https://requester.mturk.com/bulk/workers/workerID#<br />Stats shown represent all work done to date (lifetime) for the specific Requester. . <br />Click the buttons to Bonus or Block this Worker. To send a bonus you will need the ID of an assignment the Worker has submitted to you.<br />Edit a Worker’s Qualification Score. <br />Revoke a Worker’s Qualification by clicking the icon.<br />
  11. 11. Qualifications <br />Create your own Qualification Types so you can route work to specific Workers. <br />See name, ID, # of Workers with the Qualification, creation date & description <br />Qualification Types with tests – can only be created via API. <br />New Tab! Found under the “Manage” tab. <br />Dispose by clicking the icon.<br />Disposing will permanently delete the Qualification. <br />
  12. 12. Search by Requester Name<br />Enter in the Requester’s name exactly. <br />You can also click on the Requesters name in a results set to see all work currently available from that Requester. <br />Bookmark this page to always see HITs available for a specific Requester<br />
  13. 13. API Best Practices<br />We’ve made a number of improvements to the error messages from the Mechanical Turk Requester API:<br />Check your errors! <br />Use Notifications instead of the polling interfaces<br />Dispose your HITS. <br />
  14. 14. Best Practices<br />HIT Design<br />
  15. 15. Best Practices: HIT Design <br />Clear & Concise instructions<br />Add Keywords to make your HIT searchable<br />Design an efficient HIT UX<br />Listen to feedback from Workers<br />Test & Iterate<br />
  16. 16. Same Workdifferent HITs = different results<br />
  17. 17. HIT Design Principles<br />Clear statement of reward (what’s in it for me, the Worker?) <br />Built in Validation – Worker knows when they are done, Requester can prevent incomplete work<br />Detailed & collapsible instructions<br />
  18. 18. How can I get more accurate results? <br />Plurality Asking multiple Workers to do assignments, if their answers agree then the result is validated)<br />Qualifying & Training Workers<br />Bonus payments<br />Known Data Sets <br />
  19. 19. Q & A<br />Your Questions. Answered. <br />
  20. 20. What is an invalid HIT? <br />Require disclosure of the Worker's identity or e-mail address, either directly or indirectly<br />Directly or indirectly promote a site, service, or opinion <br />Violate the terms and conditions of an activity or website (for instance asking Workers to vote for something)<br />Have explicit or offensive content, for instance, nudity, but do not have the Adult Content Qualification<br />Ask Workers to solicit third parties <br />Generate "referred" site visits or click through traffic<br />Ask workers to take action to manipulate a search engine's relationship data<br />Violate copyrights <br />X<br />X<br />X<br />
  21. 21. What can you do about invalid HITs? <br />Don’t do the work. <br />Report them. <br />
  22. 22. JC Goodrich<br />Capture the Worker ID from the URL<br />API Documentation <br />2 Scenarios:<br />Submit to Turk <br />Submit to Requester<br />http://www.quora.com/Natala-Menezes/Can-you-explain-more-about-how-a-best-practice-iFrame-HIT-in-Mechanical-Turk-works<br />
  23. 23. How should I price my HIT? <br />Mechanical Turk is a Marketplace, you determine the offer for your own work – and the workers choose what work to do. <br />Test <br />Do it yourself: How long does it take? What would be a reasonable hourly wage? <br />Marketplace Test: Put in a sample HIT at different prices. How long does it take to get done? Quicker velocity = higher price, easy task = lower price <br />Many other factors also influence velocity (reputation, fun factor)<br />Combine results into one HIT – often lowers per item costs and reduces operational costs to review/approve. <br />Ex. Photo Moderation – put 5-50 photos in one HIT. <br />Compare with market pricing <br />
  24. 24. How can I attract good Workers?<br />Establish quick and fair payment. <br />Bonus your best Workers<br />Participate in the forums – Turkernation.com<br />
  25. 25. Workers: How do I handle blocks?<br />Blocks prevent Workers from working for a Requester. <br />Mechanical Turk reviews blocks as a signal of quality. <br />
  26. 26. Q & A<br />Your Questions. Answered. <br />
  27. 27. Thank you. <br />@amazonmturk<br />Facebook.com/amazonmturk<br />Mechanicalturk.typepad.com<br />

×