Artificial, Artificial Intelligence Using Amazon Mechanical Turk and .NET to Create a New Breed of Web App Jeff Barr Web Services Evangelist Amazon Web Services NGW012

Agenda Amazon Web Services Overview Artificial, Artificial Intelligence Demos Writing Mechanical Turk Applications Q&A

Amazon Web Services Overview

Artificial, Artificial Intelligence

Demos

Writing Mechanical Turk Applications

Q&A

Introduction And Background Software development background Veteran of several startups Visual Studio team at Microsoft (DHTML, XML, Web Services) 3.5 Years with Amazon Amazon Web Services Evangelist

Software development background

Veteran of several startups

Visual Studio team at Microsoft (DHTML, XML, Web Services)

3.5 Years with Amazon

Amazon Web Services Evangelist

What Is Amazon? Online Retailer Over 55 million active customer accounts Seven countries: US, UK, Germany, Japan, France, Canada, China Technology Consumer Multi-National Web Sites Vast Data Warehouse – 25 TB World-Class Logistics – 21 fulfillment centers; 9 million ft2 Technology Provider Hundreds of thousands of Amazon Associates Over 1,050,000 active seller accounts Over 150,000 software developers registered to use Amazon Web Services

Online Retailer

Over 55 million active customer accounts

Seven countries: US, UK, Germany, Japan, France, Canada, China

Technology Consumer

Multi-National Web Sites

Vast Data Warehouse – 25 TB

World-Class Logistics – 21 fulfillment centers; 9 million ft2

Technology Provider

Hundreds of thousands of Amazon Associates

Over 1,050,000 active seller accounts

Over 150,000 software developers registered to use Amazon Web Services

Web Service Architecture Operating System Libraries / DLLs / Assemblies Application Internet Web Services Application

What Is Amazon Web Services? APIs that give developers programmatic access to Amazon’s data and technology Building-block web services Web-scale infrastructure E-commerce capability Content, data, and information New business models Customer-created content

APIs that give developers programmatic access to Amazon’s data and technology

Building-block web services

Web-scale infrastructure

E-commerce capability

Content, data, and information

New business models

Customer-created content

AWS Product Family Amazon E-Commerce Service Complete access to Amazon’s product catalog Free + Associates commissions paid Amazon Historical Pricing Data warehouse access for product pricing Monthly Fee Amazon Mechanical Turk Artificial Artificial Intelligence 10% Commission Paid workforce Amazon Simple Queue Service IT building block In beta Amazon S3 Storage for the internet Charge by storage/bandwidth usage Alexa Web Information Service Data warehouse access for web crawl data 10K calls per month free, then 15 cents per 1000 calls Alexa Web Search Platform Roll your own search engine Pay for time, storage, bandwidth Alexa Top Sites Top sites by Alexa traffic rank Charges by URL

Amazon E-Commerce Service

Complete access to Amazon’s product catalog

Free + Associates commissions paid

Amazon Historical Pricing

Data warehouse access for product pricing

Monthly Fee

Amazon Mechanical Turk

Artificial Artificial Intelligence

10% Commission

Paid workforce

Amazon Simple Queue Service

IT building block

In beta

Amazon S3

Storage for the internet

Charge by storage/bandwidth usage

Alexa Web Information Service

Data warehouse access for web crawl data

10K calls per month free, then 15 cents per 1000 calls

Alexa Web Search Platform

Roll your own search engine

Pay for time, storage, bandwidth

Alexa Top Sites

Top sites by Alexa traffic rank

Charges by URL

Amazon S3 Simple storage service Storage for the internet - web service to read and write data 15 cents per Gigabyte-Month to store data 20 cents per Gigabyte to access data Private and public storage Scalable, reliable, cost-effective, and simple!

Storage for the internet - web service to read and write data

15 cents per Gigabyte-Month to store data

20 cents per Gigabyte to access data

Private and public storage

Scalable, reliable, cost-effective, and simple!

Amazon Mechanical Turk

Man Versus Machine

But Not Really

Chair Or Table?

Survey Says…

Amazon Mechanical Turk Provides Web Services API for computers to integrate artificial artificial intelligence directly into their processing by making requests of humans

Provides Web Services API for computers to integrate artificial artificial intelligence directly into their processing by making requests of humans

Put The Human In The Loop while (read (photo)) { ContainsHuman = CallMechanicalTurk(photo); if (ContainsHuman) { acceptPhoto(photo); } else { rejectPhoto(photo); } }

while (read (photo))

{

ContainsHuman = CallMechanicalTurk(photo);

if (ContainsHuman)

{ acceptPhoto(photo); } else

{ rejectPhoto(photo); }

}

How It Works www.mturk.com Workers Artificial, Artificially Intelligent Software Requester (Developer) Human Intelligence Tasks (HITs) Completed HITs Worker Qualifications

What This Means For Software Developers Solves problem of building applications that until now have not worked well without human intelligence For Businesses Efficiently, effectively and inexpensively complete millions of tasks that require human intelligence For Anyone Make money using the skills you already have whenever your schedule allows Previously unimaginable business ideas are now feasible…

For Software Developers

Solves problem of building applications that until now have not worked well without human intelligence

For Businesses

Efficiently, effectively and inexpensively complete millions of tasks that require human intelligence

For Anyone

Make money using the skills you already have whenever your schedule allows

Previously unimaginable business ideas are now feasible…

High Quality, Inexpensive, Massive Scale Image Clean-Up Which picture best represents this business?

Easy-to-use application to build HITs for Mturk Requester-oriented interface: Create and issue a series of HITs (Human Intelligence Tasks) There's no need to deal with source code or with raw XML data Free Use of HIT-Builder Functions during Beta Test period (requires Amazon Requester account)

Easy-to-use application to build HITs for Mturk

Requester-oriented interface:

Create and issue a series of HITs (Human Intelligence Tasks)

There's no need to deal with source code or with raw XML data

Building A HIT Hit- Builder.com

Building Applications With Amazon Mechanical Turk

API Status Requesters Must use APIs Must have development skills Expect some small tweaks during the beta Workers Must use mturk.com Worker API in the future

Requesters

Must use APIs

Must have development skills

Expect some small tweaks during the beta

Workers

Must use mturk.com

Worker API in the future

API Basics - #1 Use REST or SOAP, your choice Be sure to check for IsValid in each response URL encode Question and Answer XML data Include proper namespace in XML data

Use REST or SOAP, your choice

Be sure to check for IsValid in each response

URL encode Question and Answer XML data

Include proper namespace in XML data

API Basics - #2 Sign each request using your Secret Key: HMAC (Hashing Function) Service Operation Timestamp Secret Key Base 64 Encode SOAP Request REST Request URL Encode

Sign each request using your Secret Key:

System Design Process Design Great HITs Identify Worker Qualifications Determine approval strategy Build Application Deposit funds into Amazon account Load Qualifications Load HITs Approve completed Assignments Retrieve results

Design Great HITs

Identify Worker Qualifications

Determine approval strategy

Build Application

Deposit funds into Amazon account

Load Qualifications

Load HITs

Approve completed Assignments

Retrieve results

Designing A Good HIT - #1 Question elements Title Text Block List Binary Answer elements Free Text Selection (1 or multiple) Build XML using the Question Builder

Question elements

Title

Text Block

List

Binary

Answer elements

Free Text

Selection (1 or multiple)

Build XML using the Question Builder

Designing A Good HIT - #2 HIT design considerations Be clear Be verbose, if necessary Be specific about needs & expectations Leave no ambiguities Test drive and monitor before going full-bore Always have an appropriate “none of the above” choice Punctuation, capitalization, etc. Approval strategy

HIT design considerations

Be clear

Be verbose, if necessary

Be specific about needs & expectations

Leave no ambiguities

Test drive and monitor before going full-bore

Always have an appropriate “none of the above” choice

Punctuation, capitalization, etc.

Approval strategy

Identify Worker Qualifications Create a test to admit only workers with the skills or other qualifications that are needed for your particular type of HITs Qualification often looks very much like a HIT Objective (multiple choice) or subjective (free text) If necessary, use “Secret Code” qualifications Use or build on existing Qualifications

Create a test to admit only workers with the skills or other qualifications that are needed for your particular type of HITs

Qualification often looks very much like a HIT

Objective (multiple choice) or subjective (free text)

If necessary, use “Secret Code” qualifications

Use or build on existing Qualifications

Application Flow Load Qualifications Load HITs Any Completed Assignments? Retrieve and Approve Do Local Processing More Work? Y Y Done! Do Local Processing N

Load Qualifications Relevant API Calls CreateQualificationType GetQualificationRequests GrantQualification Notes Qualifications which include an Answer Key are automatically granted when passed Application must explicitly check and grant others using GrantQualification You cannot currently attach a Qualification to a worker without a prior request

Relevant API Calls

CreateQualificationType

GetQualificationRequests

GrantQualification

Notes

Qualifications which include an Answer Key are automatically granted when passed

Application must explicitly check and grant others using GrantQualification

You cannot currently attach a Qualification to a worker without a prior request

Load HITs Relevant API Calls CreateHIT - Returns HIT identifier DisableHIT – Turn it off ExtendHIT – Add additional time Notes Multiple assignments per HIT are important Auto approval time Qualification requirements can be attached to HITs Attach application data using RequesterAnnotation

Relevant API Calls

CreateHIT - Returns HIT identifier

DisableHIT – Turn it off

ExtendHIT – Add additional time

Notes

Multiple assignments per HIT are important

Auto approval time

Qualification requirements can be attached to HITs

Attach application data using RequesterAnnotation

Poll And Approve Work Relevant API Calls GetReviewableHITs GetHITs GetAssignmentsForHIT ApproveAssignment – Pay the Worker RejectAssignment – Don’t pay the Worker DisposeHIT ExtendHIT NotifyWorkers – Send e-mail notification

Relevant API Calls

GetReviewableHITs

GetHITs

GetAssignmentsForHIT

ApproveAssignment – Pay the Worker

RejectAssignment – Don’t pay the Worker

DisposeHIT

ExtendHIT

NotifyWorkers – Send e-mail notification

Retrieve Results Relevant API calls GetAssignmentsForHIT – Get the actual answers Notes GetReviewableHITs returns list of non-assignable HITs Dispose or Extend (or they’ll keep coming back)

Relevant API calls

GetAssignmentsForHIT – Get the actual answers

Notes

GetReviewableHITs returns list of non-assignable HITs

Dispose or Extend (or they’ll keep coming back)

Perform Local Processing Specific to your application Types Incremental (per Assignment) Batch (per group of related HITs) Connect application logic back to Turk If application’s needs change, Dispose of Associated HIT Adjust pricing and timing dynamically Adjust necessary Qualifications

Specific to your application

Types

Incremental (per Assignment)

Batch (per group of related HITs)

Connect application logic back to Turk

If application’s needs change, Dispose of Associated HIT

Adjust pricing and timing dynamically

Adjust necessary Qualifications

Next Steps… Find potential HITs: Within your organization Unmet or underserved market Design workflow process around the Mechanical Turk Start writing code and loading HITs Process results Get your work done!

Find potential HITs:

Within your organization

Unmet or underserved market

Design workflow process around the Mechanical Turk

Start writing code and loading HITs

Process results

Get your work done!

Q & A

Thanks!

For More Information: Mechanical Turk: www.mturk.com AWS Blog: aws.typepad.com Amazon Web Services: aws.amazon.com © 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

© 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.