Your SlideShare is downloading. ×
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
AMT: Requester UI
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

AMT: Requester UI

1,580

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,580
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Amazon Mechanical TurkRequester User Interface Guide Version
  • 2. Amazon Mechanical Turk Requester User Interface GuideAmazon Mechanical Turk: Requester User Interface GuideCopyright © 2010 Amazon Web Services LLC or its affiliates.This book was last updated 2010-11-03.
  • 3. Amazon Mechanical Turk Requester User Interface GuideTable of Contents Welcome ............................................................................................................................................................. 1 Introduction to Mechanical Turk .......................................................................................................................... 3 Overview of Mechanical Turk .................................................................................................................. 3 Advantages ................................................................................................................................... 3 Mechanical Turk Concepts ...................................................................................................................... 3 Requester ...................................................................................................................................... 4 Human Intelligence Task (HIT) ...................................................................................................... 4 Assignment .................................................................................................................................... 4 Worker ........................................................................................................................................... 4 Approval and Payment .................................................................................................................. 4 Qualification Type .......................................................................................................................... 4 Using the Requester User Interface .............................................................................................. 5 Getting Started ................................................................................................................................................... 6 Creating Your Batch of HITs ............................................................................................................................... 7 The Design Tab ....................................................................................................................................... 7 Creating a HIT Template ......................................................................................................................... 8 Publishing Your Batch of HITs .......................................................................................................................... 15 Publishing Batches ............................................................................................................................... 15 Managing Batches ............................................................................................................................................ 18 Displaying Batch Progress, Details, and Results .................................................................................. 18 Configuring the Review Results Page .................................................................................................. 20 Filtering Results .......................................................................................................................... 21 Approving and Rejecting Assignments ................................................................................................. 21 Reviewing Assignments in the RUI ............................................................................................. 22 Reviewing Assignments Offline ................................................................................................... 22 Deleting a Batch ................................................................................................................................... 24 Managing Workers ............................................................................................................................................ 26 Viewing Worker Statistics ..................................................................................................................... 26 Blocking a Worker ................................................................................................................................. 27 Unblocking a Worker ............................................................................................................................. 28 Awarding a Bonus ................................................................................................................................. 29 Assigning a Qualification to a Worker ................................................................................................... 30 Revoking a Workers Qualification ........................................................................................................ 30 Managing Worker Details Offline .......................................................................................................... 31 Managing Qualification Types ........................................................................................................................... 33 Creating a Qualification Type ................................................................................................................ 33 Viewing Existing Qualification Types ..................................................................................................... 34 Deleting Qualification Types .................................................................................................................. 35 Glossary ........................................................................................................................................................... 37 Document History ............................................................................................................................................. 39 Index ................................................................................................................................................................. 40
  • 4. Amazon Mechanical Turk Requester User Interface Guide PrerequisitesWelcome Topics • Prerequisites (p. 1) • How This Guide Is Organized (p. 1) • Amazon Mechanical Turk Resources (p. 2) Amazon Mechanical Turk gives your business access to a scalable, on-demand workforce. By helping you break your work into smaller tasks thats done by multiple Workers simultaneously, Mechanical Turk helps you get results faster and at a lower cost than was previously possible. This is the Amazon Mechanical Turk Requester User Interface Guide. It is for people who want to access Amazon Mechanical Turk functionality using a graphical user interface. This guide occasionally refers to Amazon Mechanical Turk as "Mechanical Turk," and the Amazon Mechanical Turk Requester User Interface as "RUI." All copyrights and legal protections still apply.Prerequisites This guide does not assume that you have prior knowledge of Amazon Mechanical Turk. We recommend that you visit the Resource Center to learn about best practices, user case studies, and other useful information about Amazon Mechanical Turk.How This Guide Is Organized The following table describes the organization of this guide. Information Relevant Sections Basic concepts Mechanical Turk Concepts (p. 3) Designing your HITs Creating Your Batch of HITs (p. 7) Publishing your HITs Publishing Your Batch of HITs (p. 15) Managing your batches Managing Batches (p. 18) Version 1
  • 5. Amazon Mechanical Turk Requester User Interface Guide Amazon Mechanical Turk Resources Information Relevant Sections Managing Workers Managing Workers (p. 26) Managing Qualifications Managing Qualification Types (p. 33) Whats new in this release. Document History (p. 39)Amazon Mechanical Turk Resources The following table lists related resources that can help you learn more about Amazon Mechanical Turk. Resource Description Amazon Mechanical Turk Resource Contains links to helpful guides, videos, webinars, case Center studies, and other tools to help you get started and manage your HITs. Amazon Mechanical Turk FAQ Includes important information about your account and taxes. Participation Agreement Consists of the terms and conditions for using the Mechanical Turk site. Privacy Notice Describes the information Amazon gathers from users. Contact Us Provides email support for using the Requester User Interface. Version 2
  • 6. Amazon Mechanical Turk Requester User Interface Guide Overview of Mechanical TurkIntroduction to Mechanical Turk Topics • Overview of Mechanical Turk (p. 3) • Mechanical Turk Concepts (p. 3) This introduction to Mechanical Turk provides a detailed summary of this user interface. After reading this section, you should have a good idea what the Amazon Mechanical Turk Requester User Interface (RUI) offers and how it can fit in with your business.Overview of Mechanical Turk Mechanical Turk provides a marketplace for work and gives you access to Workers around the world and around the clock. Mechanical Turk enables you to complete a variety of tasks, including data categorization, moderation, verification, and tagging. Advantages The major advantages of Mechanical Turk are: • On demand workforce—Workers are available whenever you need them • Scalable workforce—You can use one or thousands of Workers to complete your assignments • Qualified workforce—You can qualify Workers to work on your assignments to get the best results • Pay only for satisfactory work—You don’t pay Workers or Mechanical Turk fees until you accept their assignments; if you reject their work, they do not get paidMechanical Turk Concepts Topics • Requester (p. 4) • Human Intelligence Task (HIT) (p. 4) • Assignment (p. 4) • Worker (p. 4) Version 3
  • 7. Amazon Mechanical Turk Requester User Interface Guide Requester • Approval and Payment (p. 4) • Qualification Type (p. 4) • Using the Requester User Interface (p. 5)This section describes the concepts and terminology you need to understand to use Mechanical Turkeffectively.RequesterA Requester creates tasks on Mechanical Turk for Workers to work on. As a Requester, you use theRequester User Interface (RUI) to create tasks, check the status of your tasks, and accept or reject workperformed on tasks. Workers see your account name (specified by your Amazon.com account) whenthey view and accept your tasks.Human Intelligence Task (HIT)A Human Intelligence Task (HIT) is a single, self-contained task a Requester creates on Mechanical Turk,for example, "Identify the color of the car in the photo."AssignmentYou can assign many Workers to work on the same HIT. This is a useful way of getting agreement onwhat is correct. A Worker can only accept a HIT once and can only submit one assignment per HIT. Thisguarantees that multiple Workers must complete a HIT that has multiple assignments.If a Worker fails to complete an assignment before the time frame specified (i.e., the Worker abandonsthe HIT), or if the Worker chooses not to complete it after accepting it (i.e., the Worker returns the HIT),the assignment becomes available for other Workers to work on.WorkerA Worker is a person who completes assignments. Workers use the Mechanical Turk web site(http://mturk.amazon.com/) to find assignments to work on, submit responses, and manage their account.A Requester sees the Workers account ID for every assignment submitted.Approval and PaymentWorkers submit responses for assignments. If you approve their work, Mechanical Turk transfers the HITreward from your Mechanical Turk account to their Amazon Payments accounts. Workers dont get paidif you reject their work.When you post a batch of HITs, you agree to approve or reject work by a specified deadline. If thatdeadline passes, Mechanical Turk automatically approves the assignments and pays the Workers whosubmitted work.Before you can post your batch of HITs, you must have enough money in your account to pay for all ofthe work. You can deposit money into your Mechanical Turk account at any time using the Requesterweb site (http://Requester.mturk.amazon.com/).Qualification TypeMechanical Turk provides System Qualification Types that automatically keep track of a Workers accountstatistics and attributes. For example, one System Qualification Type is the Workers approval rating for Version 4
  • 8. Amazon Mechanical Turk Requester User Interface Guide Using the Requester User Interfaceall approved and rejected assignments. You can use System Qualification types to control who can andcannot work on your HITs. For example, you can require that Workers have a 95% or greater approvalrating to work on your HITs.You can also create your own Qualification Types and assign them to Workers. You can stipulate thatonly Workers assigned your Qualification Type can work on your HITs. Each Qualification Type you createincludes a name, description, and a score, which is an integer from 0 to 100. You use the score todifferentiate Workers that have the same Qualification Type.Using the Requester User InterfaceThe Requester User Interface (RUI) makes it easy to create a HIT template, publish HITs, manage batchresults, and manage Workers. The following graphic and process show the basic workflow.Using the RUI1 Define the properties and design the layout of the HIT template.2 Publish your batch of HITs.3 Approve or reject the work done on your HITs. You can download the results, review them offline, and republish rejected HITs.4 View a Worker’s approval rating, award bonuses, assign qualifications, or block Workers from working on your HITs. Version 5
  • 9. Amazon Mechanical Turk Requester User Interface GuideGetting Started To get started, you must create a Mechanical Turk account, or sign in if you have one. To use the Requester User Interface 1. Go to the Amazon Mechanical Turk Resource Center and do one of the following. To... Do This... Create a Mechanical Turk account Click Sign Up Now and follow the on-screen instructions. Sign in using your Mechanical Turk account Click Sign In and follow the on-screen instructions. 2. Click the Design tab. Version 6
  • 10. Amazon Mechanical Turk Requester User Interface Guide The Design TabCreating Your Batch of HITs Topics • The Design Tab (p. 7) • Creating a HIT Template (p. 8) This section shows you how to create a batch of HITs.The Design Tab You need to create a HIT template to create a batch of HITs.You can create a HIT template by customizing an existing template. The Design tab lists available templates in two tables. The Your HIT Templates table lists templates you created. The Sample HIT Templates table lists templates supplied by Mechanical Turk. Version 7
  • 11. Amazon Mechanical Turk Requester User Interface Guide Designing Your HIT Template Designing Your HIT Template A HIT template is an HTML page that presents HITs to Workers. There are two kinds HIT templates; ones that: • Dont contain variables In this case, all Workers see the same HTML page. You might use this to create a survey. • Contain variables so that every HIT has the same format but each HIT varies For example, you might ask Workers to provide key words for photos and each HIT would have a different photo. If your HIT template contains variables, you must provide a data file that contains the values for the variables. The column headings in the data file must match the variable names in the HIT template. One row of data in the data file creates one HIT. The values can be text, or the URLs of images or videos. The RUI merges the HIT template and the values in the data file to create a batch of HITs.Creating a HIT Template The following process describes the steps you take to create a HIT template. Creating a HIT Template 1 Define the HIT template properties. These properties are HIT metadata, for example, how long Workers can work on a HIT. 2 Design the HTML HIT template layout. Take into consideration the variables in your HIT data file. In this step, you create the template the RUI uses to create the HITs in the batch. 3 Preview the HIT template. Version 8
  • 12. Amazon Mechanical Turk Requester User Interface Guide Creating a HIT TemplateTo create a HIT template1. In the HIT Sample Templates table, choose one of the provided templates by clicking one of the Start with this template buttons. The following image shows the HIT Sample Templates table. The following image shows the RUI display for the Image Tagging HIT template. Version 9
  • 13. Amazon Mechanical Turk Requester User Interface Guide Creating a HIT Template2. Use the following table to enter the HITs properties. Field Description Template Name Enter a template name. The RUI does not display this name to Workers. If you use a supplied template, the RUI fills in this field. Title Enter the name of the HIT. Make sure to be specific, such as "Tag landmark images" instead of "Tag photos." The RUI displays this title to Workers. Description Describe the HIT. The search mechanism searches through this description so use words you think would help Workers find your HITs. Keywords Enter a comma-separated list of words that Workers can use to find your HIT. Include your Requester name. Time Allowed Per Specify the maximum amount of time a Worker can work on an assignment. The Assignment RUI withdraws the assignment from the Worker once this time expires so others can work on it. The countdown starts once the Worker accepts the assignment. HIT Expires in Specify how long Workers can accept HITs in the batch. Workers cant accept HITs in the batch once this time expires. Workers can finish working on assignments they previously accepted even though the batch is no longer available for others to work on. Worker must meet Use the drop-down menu to specify Qualification Types the Worker must have the following to qualify for working on the batch. You can use System Qualification Types criteria.. provided by Mechanical Turk or custom Qualification Types you create. Add another Click this button to specify additional qualifications. criteria Required for Select this check box to require that a Worker have the specified qualifications preview to preview these HITs. Reward per Specify how much youll pay the Worker if you approve an assignment. assignment Number of Specify the number of Workers you want to work on each HIT. One assignment assignments per per HIT means that only one Worker works on a HIT. You might want multiple HIT Workers to work on a HIT to see if there is agreement, which can increase your trust in the results. A Worker can only work on one assignment per HIT. If a HIT has multiple assignments, the RUI guarantees that multiple Workers will work on the HIT. Results are Specify when the RUI will automatically approve HITs you havent reviewed. automatically This limit ensures that Workers get paid in timely manner. approved in3. Click Design Layout. The Design Layout tab appears. Version 10
  • 14. Amazon Mechanical Turk Requester User Interface Guide Creating a HIT Template4. Use the controls in the formatting toolbar to edit the text on the page. The icons are similar to those used in most word processors. If you hover your mouse over an icon, the RUI displays a description of it. To author in HTML, click Edit HTML Source.5. Choose the names for the variables in your template. If you use a data file to populate your HITs, you must make the variable names match the column headings in your data file. For example, if the column heading is continent, ${continent} is the variable name youd use in your HIT template. You use a dollar sign ($) and curly brackets to create a text variable. Each row in your HIT data file must contain all of the values for one HIT, as shown in the following figure. Version 11
  • 15. Amazon Mechanical Turk Requester User Interface Guide Creating a HIT Template6. Enter variables for images and videos in the template. If your template includes variables for images or videos, you must edit the HTML directly by clicking Edit HTML Source. Refer to the following table to see how to create variables for images and videos. Variable Format image HTML image tag, <img>, where the alt attribute defines the name of the variable, for example: <img width="200" height="200" alt="imagevariableName" style="margin-right: 10px;" src="${image_url}" /> video HTML object tag, <object>, where the name attribute defines the name of the variable, for example: <object width="425" height="344"><param name="movie" value="${video_url}"></param><param name="allowFullScreen" value="true"></param><embed src="${video_url}" type="application/x-shockwave-flash" allowfullscreen="true" width="425" height="344"></embed></object>7. Create your data file. Separate the values in the HIT data file with commas and save it using the .csv format. Many spreadsheet applications, including Microsoft Excel, can save files as .csv. One row of data should exactly match the number of values needed in each HIT. The order in which you present the variables in the HIT template does not need to match the order of columns in the .csv file. Version 12
  • 16. Amazon Mechanical Turk Requester User Interface Guide Creating a HIT Template Note The RUI doesnt support line breaks between cells nor does it support "r" as a line break character. Apple Macintosh computers insert this character when they convert a Microsoft Excel table into a .csv file. If your HITs contain images or videos, you must include links to them in the data file, and the images and videos must be publicly accessible. The RUI does not provide a tool for uploading images or videos. Consider using one of the publicly-available tools, such as Bucket Explorer, to upload your images into Amazon S3.8. Create answer fields in your template where Workers can enter answers. You must include the HTML name attribute in the answer field definition because the name becomes a column heading in the results table. In the following example, the answer field name is Landmark and its size is 25. <input type="text" name="Landmark" id="Field1" size="25" /></td> The RUI returns results in a table stored in a comma-separated-value file format with the extension, .csv. The column headings are the variable names prepended by Input, and the answer field names, prepended by Answer. For example, if the variable is continent, the column heading is Input.continent. The following is a sample results table. One row in the results table contains the responses for one assignment.9. Save the HTML of your HIT template. Mechanical Turk deletes the HIT templates if you dont use them for 120 consecutive days. If you need to access your template longer, we recommend that you copy the HTML and save it on your own system.10. Click Preview and Finish. The RUI displays the HIT Preview page. Version 13
  • 17. Amazon Mechanical Turk Requester User Interface Guide Creating a HIT Template11. Click Finish if youre satisfied with the HIT properties and template.You have set the HIT properties for your HIT template and designed the HTML page that Workers willsee. Now, you need to publish the HIT template to create the batch of HITs and make them available forWorkers to work on. Version 14
  • 18. Amazon Mechanical Turk Requester User Interface Guide Publishing BatchesPublishing Your Batch of HITs Publishing your HITs on the Mechanical Turk web page gives Workers the opportunity to work on them. If youre publishing a HIT without variables, you only need to publish the HTML page you created in the previous section. If your HIT template contains variables, you must upload a .csv data file that contains the values for the variables in each HIT.Publishing Batches Use the following procedure to publish your batch of HITs. Note The following procedure assumes that you will upload a HIT data file. To publish a batch of HITs 1. Click the Publish tab. The RUI displays the Select HIT Template page. 2. Click Select beside the HIT template you want to publish. The RUI displays the Upload Input Data page if your HIT template contains variables. If it doesnt, the RUI skips to task 3, Preview. Version 15
  • 19. Amazon Mechanical Turk Requester User Interface Guide Publishing Batches3. Click Browse to locate the .csv data file that you want to upload, and click Upload. The RUI displays the HIT Preview page. Version 16
  • 20. Amazon Mechanical Turk Requester User Interface Guide Publishing Batches4. Carefully review your HITs so that you catch mistakes before publishing. a. To see the next HIT, click Next HIT. You can preview up to 200 HITs in your data file. Note To use a different, previously-loaded .csv file, click Select a Different Input File. b. Click Next HIT after previewing your HITs. The RUI displays the Batch Summary page. This page shows the total amount you will pay Workers and Mechanical Turk if you approve all of the assignments. The following table explains the sections of the Batch Summary page. Section Description Batch Contains the values you set on the Design tab for the batch properties, including the Properties number of days the batch can exist before expiring, and the number of days you have to approve assignments before the RUI automatically approves the assignments. HITs Calculates the number of assignments per batch. In this case, there are three assignments per HIT and three HITs for a total of nine assignments. Cost Calculates the cost of the batch assuming you approve all assignments. The total cost is the number of assignments multiplied by the price per assignment plus the Mechanical Turk fee. You must have the total cost in your account before you can publish the HIT. If you dont, the RUI asks you to add money to your account.5. Click Publish HITs to publish the batch of HITs. Version 17
  • 21. Amazon Mechanical Turk Requester User Interface Guide Displaying Batch Progress, Details, and ResultsManaging Batches Topics • Displaying Batch Progress, Details, and Results (p. 18) • Configuring the Review Results Page (p. 20) • Approving and Rejecting Assignments (p. 21) • Deleting a Batch (p. 24) On the Manage tab, you can manage your batch results, the Workers who work on your batches, and your Qualification Types. On the Batches page on the Manage tab, you can view: • The progress of your batch • The batch details • The batch resultsDisplaying Batch Progress, Details, and Results The following procedure describes how to display information related to your batches. To display the batch progress, details, and results 1. Click Batches on the Manage tab. The RUI displays headings for batches in different states. 2. Click the batch state you want to view. The RUI displays batches in that state. Version 18
  • 22. Amazon Mechanical Turk Requester User Interface Guide Displaying Batch Progress, Details, and Results The progress bar shows two quantities: • % submitted—Percentage of completed HITs • % published—Percentage of published HITs in your batch Most batches publish quickly, but extremely large batches can take several minutes to publish.3. Click the name of the batch you want to view. The RUI displays the Batch Details page. This page contains all of the batch properties you entered when creating this batch.4. Click Results to see the submissions for the assignments in the batch. The RUI displays the Review Results page. Version 19
  • 23. Amazon Mechanical Turk Requester User Interface Guide Configuring the Review Results Page This page shows detailed information about the HIT results, such as the Worker who completed the HIT, the Workers approval rating, and the Workers submission for your HITs.Configuring the Review Results Page The RUI enables you to display the columns you want in the order you want on the Review Results page. To configure the Review Results page 1. Click Customize View on the Batches page. The RUI displays the Configure Data Results page. 2. Do one or more of the following. To.. Do This.. Add fields listed in the Drag field names from the Available fields list into the Show these fields results in this order list. Version 20
  • 24. Amazon Mechanical Turk Requester User Interface Guide Filtering Results To.. Do This.. Remove fields listed in Drag field names from the Show these fields in this order list into the the results Available fields list. Change the order of Drag the fields up or down in the Show these fields in this order list. the fields in the results 3. Click Apply Settings. Filtering Results The RUI enables you to display a subset of the results. You can filter out results based on the amount of time it took to do the assignment, and on the state of the result (submitted, approved, or rejected). By default, the RUI displays only submitted work. To filter the results 1. Click Filter Results on the Review Results page. The RUI displays the Filter Results page. 2. To show assignments in a specified state, select the Status Filter check box, and select the state in the drop-down menu that you want to view. 3. Select the Velocity Filter check box and specify a number of seconds if you only want to show assignments completed in a specified amount of time or less. 4. Click Apply Filters.Approving and Rejecting Assignments Topics • Reviewing Assignments in the RUI (p. 22) • Reviewing Assignments Offline (p. 22) A Requester approves or rejects assignments that Workers submit. When you approve an assignment the Worker gets paid; when you reject an assignment the Worker does not get paid. You can approve results individually or all at once. Also, you can sort the table based on the Worker ID so you can review the results from a specific Worker all at once. Version 21
  • 25. Amazon Mechanical Turk Requester User Interface Guide Reviewing Assignments in the RUIReviewing Assignments in the RUIIts convenient to use the RUI to review relatively small numbers of assignments.To approve or reject results using the RUI1. On the Review Results page, do one of the following. To.. Do This.. Approve individual results Click the check box in the row(s) of results you want to approve, and click Approve. Reject individual results Click the check box in the row(s) of results you want to reject, and click Reject. Approve all results in the batch Click Approve All, or click the check box in the column heading of check boxes and click Approve. Reject all results in the batch Click the check box in the column heading of check boxes and click Reject.2. If you want to sort the list by Worker, click the Worker ID column heading. Tip You can easily approve or reject all of a Workers work. Over time you might find a Worker who consistently submits excellent work and therefore has a very high approval rating. In that case, you could approve all of the Workers work without reviewing all of it. Mechanical Turk processes payments several times a day so there is always a small delay between approval and payment. When there is a delay, Workers see HITs as Approved-pending payment in their dashboard. The dashboard displays Paid for processed payments.Reviewing Assignments OfflineIf a batch is large, its often easier to download batch results, review them using another application, suchas Microsoft Excel, and then upload the revised file. Note Results are available for 120 days after you approve or reject them. If you need to access the results longer than that, use the following procedure to download and archive them.To download and review assignments offline1. On the Review Results page, click Download CSV. The RUI displays the following dialog box. Version 22
  • 26. Amazon Mechanical Turk Requester User Interface Guide Reviewing Assignments Offline2. Click the word, here. The RUI displays the following dialog box.3. In this dialog box, do one of the following. To.. Do This.. To open and view the Select Open with and choose the application from the pull down list that results you want to use to view the results. The application you select must be able to display .csv files. To save the results Select Save to Disk.4. Click OK and open the results file with a .csv compatible application, such as Microsoft Excel.5. For each result, enter an X in the Approve column, or a reason to reject the result in the Reject column.6. Save the spreadsheet as a .csv file. Version 23
  • 27. Amazon Mechanical Turk Requester User Interface Guide Deleting a Batch Now that youve reviewed the results, you must upload your file so Mechanical Turk can approve or reject work based on the changes you made to the file. To upload a .csv file into the RUI 1. On the Review Results page, click Upload CSV. The RUI displays the Review Results Offline dialog box. 2. Click Browse, select the .csv file you saved, and click Upload CSV. The following dialog box appears. 3. To republish rejected assignments, select the Republish rejected assignment(s) for other Workers to complete check box. 4. Click Yes to confirm your choices. Note Some versions of Microsoft Excel do not display international characters by default. If your HIT title, description, or HIT results contain international characters, you must follow the instructions for your version of Microsoft Excel to import or activate international characters.Deleting a Batch If the batch you published isnt working the way youd like, you can delete it. To delete a batch 1. On the Batches page, click Delete on the batch you want to delete. Version 24
  • 28. Amazon Mechanical Turk Requester User Interface Guide Deleting a Batch The Delete Batch dialog box appears.2. Click Yes. Note It can take several minutes to delete a batch. All Workers who accepted assignments before you deleted the batch can continue working on them. The batch will not be completely deleted until all assignments accepted by Workers have either been returned, submitted, or abandoned. Version 25
  • 29. Amazon Mechanical Turk Requester User Interface Guide Viewing Worker StatisticsManaging Workers Topics • Viewing Worker Statistics (p. 26) • Blocking a Worker (p. 27) • Unblocking a Worker (p. 28) • Awarding a Bonus (p. 29) • Assigning a Qualification to a Worker (p. 30) • Revoking a Workers Qualification (p. 30) • Managing Worker Details Offline (p. 31) The Manage tab in the RUI enables Requesters to track Worker performance and take appropriate actions, including blocking Workers, awarding bonuses, and assigning qualifications. Requesters can view the following details on all of the Workers who have worked for them: • Worker’s approval rating on your HITs • Worker’s blocked status • Worker’s qualification score for up to five of your Qualification Types Mechanical Turk automatically keeps track of the number of assignments youve approved and rejected for each Worker. You might choose to block a Worker whose assignments you consistently rejected in the past. You might choose to give a bonus to Workers whose work is consistently excellent. The following procedures begin on the Manage tab on the Workers page.Viewing Worker Statistics Mechanical Turk enables you to view a Workers statistics, which characterize what the Worker is good at. To view a Workers statistics 1. Click Workers. The Manage Workers page appears. Version 26
  • 30. Amazon Mechanical Turk Requester User Interface Guide Blocking a Worker The Block Status column can have the following values: • Never Blocked—Worker has never been blocked by you • Blocked—Worker is not allowed to work for you • Unblocked—Worker was blocked by you at one time, but no longer is 2. To take a specific action on an individual Worker, click a Worker ID. The RUI displays the Manage Individual Worker page. On this page you can view the Workers approval rating, as well as the number of assignments you approved and rejected.Blocking a Worker If Workers arent performing to your standards, you can block them from working on your HITs. To block a Worker 1. On the Worker page, double click the Worker you want to block. The Manage Individual Worker page appears. Version 27
  • 31. Amazon Mechanical Turk Requester User Interface Guide Unblocking a Worker 2. Click Block Worker. The Block Worker dialog box appears. 3. Enter a reason for blocking the Worker, and then click Block. The RUI sends this message to the Worker.Unblocking a Worker If you mistakenly blocked a Worker, you can unblock them. To unblock a Worker 1. On the Worker page, double click the Worker you want to unblock. The Manage Individual Worker page appears. 2. Click Unblock Worker. Version 28
  • 32. Amazon Mechanical Turk Requester User Interface Guide Awarding a Bonus The Unblock Worker dialog box appears. 3. Enter a reason for unblocking the Worker, and click Unblock. The RUI sends this message to the Worker.Awarding a Bonus You can award bonuses to Workers to encourage them to keep working for you. To give a Worker a bonus 1. On the Worker page, double click the Worker you want to award a bonus. The Manage Individual Worker page appears. 2. Click Bonus Worker. The Bonus Worker page appears. Version 29
  • 33. Amazon Mechanical Turk Requester User Interface Guide Assigning a Qualification to a Worker 3. Enter the amount of the bonus, the ID of the assignment worked on, and the reason for the bonus, and then click Pay Bonus Now.Assigning a Qualification to a Worker This section shows how to assign a Worker a Qualification Type. For information about creating a Qualification Type, see How to Manage Qualification Types. To assign a Qualification Type to a Worker 1. On the Manage Individual Worker page, click Assign Qualification Type. The RUI displays the Assign Qualification Type page. 2. Select the check box(es) next to the Qualification Type you want to assign to the Worker. The Score text box appears beneath each selected Qualification Type. 3. Enter a score (0 to 100) and click Assign. Use scores to differentiate Workers that have the same Qualification Type. The Workers qualification appears at the bottom of the window. The edit link in the Score column enables you to change the qualification score.Revoking a Workers Qualification The following procedure revokes a Workers qualification. To revoke a Workers qualification • On the Manage Individual Worker page, click the X next to the Qualification Type you want to revoke. Version 30
  • 34. Amazon Mechanical Turk Requester User Interface Guide Managing Worker Details OfflineManaging Worker Details Offline If you have a large number of Workers, its easier to manage them offline using another application, such as Microsoft Excel. This section shows how to download Worker information, edit it, and upload it into the RUI. To download and modify Worker details 1. On the Manage Workers page, click Download CSV. The RUI displays the Download Workers Results page. 2. Click the word, here to download the Worker data file. The Worker data downloads and opens in Microsoft Excel. The CURRENT-QualName column shows the Workers current qualification score. If the cell is blank, you havent assigned the Qualification Type to the Worker. The far, right column, CURRENT-BLOCK STATUS, shows the Workers block status. 3. To update the values, do one or more of the following: • Indicate which Qualification Type to assign a Worker by putting a Qualification score in the UPDATE-QualName column, for example, UPDATE-Good Tagger. Qualification scores must be 0 to 100, inclusive. • To revoke the Qualification Type, enter Revoke in the UPDATE column for your Qualification Type. • Block or unblock a worker by putting Block or Unblock in the UPDATE-BlockStatus column. 4. Save the .csv file, and click Workers. The Manage Workers page appears. 5. On the Manage Workers page, click Upload CSV. The Manage Workers Offline window appears. Version 31
  • 35. Amazon Mechanical Turk Requester User Interface Guide Managing Worker Details Offline6. Click Browse to find the .csv file you saved, and then click Upload CSV. The Offline Processing window appears.7. Click Yes to confirm that you would like to save the changes for your Workers. Version 32
  • 36. Amazon Mechanical Turk Requester User Interface Guide Creating a Qualification TypeManaging Qualification Types Topics • Creating a Qualification Type (p. 33) • Viewing Existing Qualification Types (p. 34) • Deleting Qualification Types (p. 35) You can create your own Qualification Types, or use the ones supplied by Mechanical Turk. Mechanical Turk provides System Qualification Types that automatically keep track of a Workers account statistics and attributes. You can use System Qualification types to control who can and cannot work on your HITs. For example, you can require that Workers have a 95% approval rating or greater to work on your HITs. You can create new Qualification Types to select Workers based on any criteria you want.You can assign a Qualification Type and a score to Workers who work for you. Then when creating a HIT, you can specify the Qualification Type and the minimum score a Worker must have to be eligible to work on your HITs. Note The RUI does not support Qualification Tests that a Worker must take to achieve a qualification. Use the Mechanical Turk APIs or the command line tools to use tests. For more information, go to http://aws.amazon.com/documentation/mturk/. The following procedures start on the Manage tab on the Qualification Types page.Creating a Qualification Type The following procedure shows you how to create your own Qualification Type. To create a new Qualification Type 1. Click Create New Qualification Type. The Create New Qualification Type dialog box appears. Version 33
  • 37. Amazon Mechanical Turk Requester User Interface Guide Viewing Existing Qualification Types 2. In the Friendly Name text box, enter a name that describes the Qualification Type. 3. In the Description text box, enter a description of the Qualification Type and click Create. The new Qualification Type appears in the list of Qualification Types on the Manage Qualification Types page. There is a short delay before the new Qualification Type appears in the list. You can refresh your browser to update the list. To assign Workers your new Qualification Type, see Assigning a Qualification Type to a Worker (p. 30).Viewing Existing Qualification Types The following procedure shows you the Qualification Types you created. To view Qualification Types • Click Qualification Types from the Manage tab. The Qualification Types page appears and shows you all of your Qualification Types. Version 34
  • 38. Amazon Mechanical Turk Requester User Interface Guide Deleting Qualification TypesDeleting Qualification Types This section explains how to delete Qualification Types that you created. 1. Click Qualification Types on the Manage tab. The Qualification Types page appears. 2. Click the X next to the Qualification Type you want to delete. 3. When the RUI asks you to confirm the deletion, confirm it. Version 35
  • 39. Amazon Mechanical Turk Requester User Interface Guide Deleting Qualification TypesThere is a short delay before the Qualification Type disappears from the list. You can refresh yourbrowser to update the list.When you delete a Qualification Type the RUI automatically removes it from all of your Workers andHIT templates. The RUI does not remove the Qualification Type from HITs that Workers are workingon. Version 36
  • 40. Amazon Mechanical Turk Requester User Interface GuideGlossary approval If a Worker’s response satisfies your HIT’s task, you approve the assignment. When you approve an assignment Mechanical Turk transfers the HIT reward from your Mechanical Turk account to the Workers Amazon Payments account. assignment A HIT can have many assignments, that is, you can assign many Workers the same task. This is a useful way of getting agreement on what is correct. A Worker can only accept a HIT once and can only submit one assignment per HIT.This guarantees that multiple Workers must complete a HIT that has multiple assignments. batch A batch is one or more HITs created and published using the Mechanical Turk Requester User Interface. HIT A Human Intelligence Task (HIT) is a task that a Requester creates on Mechanical Turk. A HIT represents a single, self-contained task, for example, "Identify the color of the car in the photo." keyword A keyword is a word that Mechanical Turk Workers use to search for a HIT. Qualification Type Mechanical Turk provides System Qualification Types that keep track of a Workers account statistics and attributes. You can use System Qualification types to control who can or cannot work on your HITs. For example, you can require that Workers have a 95% approval rating or greater to work on your HITs. You can also create your own Qualification Types and assign them to Workers. You could stipulate that only Workers assigned your Qualification Type can work on your HITs. Each Qualification Type you create includes a name, description, and a score. Requester A Requester creates tasks on Mechanical Turk for Workers to work on. As a Requester, you use the Requester User Interface (RUI) to create tasks, check the status of your tasks, and accept or reject submitted tasks. Workers see your account name (specified by your Amazon.com account) when they view and accept your tasks. Version 37
  • 41. Amazon Mechanical Turk Requester User Interface GuideWorkers A Worker is a person who completes assignments. A Worker uses the Mechanical Turk web site (http://mturk.amazon.com/) to find HITs to work on, submit responses for HITs, and manage his or her account. A Requester sees the Workers account ID for every assignment submitted. Version 38
  • 42. Amazon Mechanical Turk Requester User Interface GuideDocument History The following table describes the important changes since the last major release of this guide. This guide was last updated on 03 November 2010. Change Description Release Date Manage Batches The Manage tab now has a page called Manage 2010-10-04 Batches. On that page, you can approve and reject assignments, import and export results, view batch details, and delete a batch. For more information, see Creating Your Batch of HITs (p. 7). Manage Workers The Manage tab now has a page called Manage 2010-10-04 Workers. On that page, you can block Workers, give them bonuses, and assign them a Qualification Type and score. For more information, see Managing Workers (p. 26). Manage Qualification The Manage tab now has a page called Manage 2010-10-04 Types Qualification Types. On that page, you can view the details of Qualification Types and create new ones. For more information, see Managing Qualification Types (p. 33). Bonus Workers Now you can award a bonus while youre reviewing a 2010-02-25 Workers results. For more information, see Managing Batches (p. 18). Reject and Republish Now you can reject work and republish an assignment 2010-02-25 while youre reviewing a Workers results. For more information, see Managing Batches (p. 18). Version 39
  • 43. Amazon Mechanical Turk Requester User Interface Guide PIndex publish batch, 15A Qadvantage, 3 qualificationapprove assignment, 21 assign, 30approve payment, 4 revoke, 30assign qualification, 30 Qualification Type, 4, 33assignment, 4 create, 33 approve, 21 delete, 35award, 29 view, 34B Rbatch reject assignment, 21 delete, 24 related resources, 2 manage, 18 Requester, 4 publish, 15 results results, 18 batch, 18block a Worker, 27 configure page, 20bonus, 29 download, 22 filter, 21 upload reviewed, 24C reviewconcepts, 4 results, 22 revoke qualification, 30D RUI, 5data file, 12 review, 22delete a batch, 24delete Qualification Type, 35 UDesign tab, 7 unblock a worker, 28document history, 39F W Worker, 4filter results, 21 Worker details, 26Gget started, 6glossary, 37guide organization, 1HHIT, 4 properties, 10 template, 8 create, 8Mmanage batches, 18manage Workers, 26 offline, 31Ooffline, Workers, 31organization of guide, 1 Version 40

×