Artificial, Artificial Intelligence Using Amazon Mechanical Turk and .NET to Create a New Breed of Web App Jeff Barr Web S...
Agenda <ul><li>Amazon Web Services Overview </li></ul><ul><li>Artificial, Artificial Intelligence </li></ul><ul><li>Demos ...
Introduction And Background <ul><li>Software development background </li></ul><ul><li>Veteran of several startups </li></u...
What Is Amazon? <ul><li>Online Retailer </li></ul><ul><ul><li>Over 55 million active customer accounts </li></ul></ul><ul>...
Web Service Architecture Operating System Libraries / DLLs / Assemblies Application Internet Web Services Application
What Is Amazon Web Services?  <ul><li>APIs that give developers programmatic access to Amazon’s data and technology </li><...
AWS Product Family <ul><li>Amazon E-Commerce Service </li></ul><ul><ul><li>Complete access to Amazon’s product catalog </l...
Amazon S3 Simple storage service <ul><li>Storage for the internet - web service to read and write data </li></ul><ul><li>1...
Amazon Mechanical Turk
Man Versus Machine
But Not Really
Chair Or Table?
Survey Says…
Amazon Mechanical Turk <ul><li>Provides Web Services API for computers to integrate artificial artificial intelligence dir...
Put The Human In The Loop <ul><li>while (read (photo)) </li></ul><ul><li>{ </li></ul><ul><li>  ContainsHuman = CallMechani...
How It Works www.mturk.com Workers Artificial, Artificially  Intelligent Software Requester (Developer) Human  Intelligenc...
What This Means <ul><li>For Software Developers </li></ul><ul><li>Solves problem of building applications that until now h...
High Quality, Inexpensive, Massive Scale Image Clean-Up Which picture best represents this business?
<ul><li>Easy-to-use application to build HITs for Mturk </li></ul><ul><li>Requester-oriented interface:   </li></ul><ul><u...
Building A HIT Hit- Builder.com
Building Applications With Amazon Mechanical Turk
API Status <ul><li>Requesters </li></ul><ul><ul><li>Must use APIs </li></ul></ul><ul><ul><li>Must have development skills ...
API Basics - #1 <ul><li>Use REST or SOAP, your choice </li></ul><ul><li>Be sure to check for IsValid in  each response </l...
API Basics - #2 <ul><li>Sign each request using your  Secret Key: </li></ul>HMAC (Hashing Function) Service Operation Time...
System Design Process <ul><li>Design Great HITs </li></ul><ul><li>Identify Worker Qualifications </li></ul><ul><li>Determi...
Designing A Good HIT - #1 <ul><li>Question elements </li></ul><ul><ul><li>Title </li></ul></ul><ul><ul><li>Text Block </li...
Designing A Good HIT - #2 <ul><li>HIT design considerations </li></ul><ul><ul><li>Be clear </li></ul></ul><ul><ul><li>Be v...
Identify Worker Qualifications <ul><li>Create a test to admit only workers with the skills or other qualifications that ar...
Application Flow Load Qualifications Load HITs Any Completed  Assignments? Retrieve and  Approve Do Local Processing More ...
Load Qualifications <ul><li>Relevant API Calls </li></ul><ul><ul><li>CreateQualificationType </li></ul></ul><ul><ul><li>Ge...
Load HITs <ul><li>Relevant API Calls </li></ul><ul><ul><li>CreateHIT - Returns HIT identifier </li></ul></ul><ul><ul><li>D...
Poll And Approve Work <ul><li>Relevant API Calls </li></ul><ul><ul><li>GetReviewableHITs </li></ul></ul><ul><ul><li>GetHIT...
Retrieve Results <ul><li>Relevant API calls </li></ul><ul><ul><li>GetAssignmentsForHIT – Get the  actual answers </li></ul...
Perform Local Processing <ul><li>Specific to your application </li></ul><ul><li>Types </li></ul><ul><ul><li>Incremental (p...
Next Steps… <ul><li>Find potential HITs: </li></ul><ul><ul><li>Within your organization </li></ul></ul><ul><ul><li>Unmet o...
Q & A
Thanks!
For More Information: Mechanical Turk:  www.mturk.com AWS  Blog:  aws.typepad.com Amazon Web Services:  aws.amazon.com © 2...
© 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes n...
Upcoming SlideShare
Loading in...5
×

Artificial Artificial Intelligence: Using Amazon Mechanical Turk and .NET to Create a…

1,861

Published on

Amazon Mechanical Turk is a new Web service that allows .NET software developers to incorporate the power of human decision-making into their automated software systems

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,861
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
72
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Artificial Artificial Intelligence: Using Amazon Mechanical Turk and .NET to Create a…

    1. 1. Artificial, Artificial Intelligence Using Amazon Mechanical Turk and .NET to Create a New Breed of Web App Jeff Barr Web Services Evangelist Amazon Web Services NGW012
    2. 2. Agenda <ul><li>Amazon Web Services Overview </li></ul><ul><li>Artificial, Artificial Intelligence </li></ul><ul><li>Demos </li></ul><ul><li>Writing Mechanical Turk Applications </li></ul><ul><li>Q&A </li></ul>
    3. 3. Introduction And Background <ul><li>Software development background </li></ul><ul><li>Veteran of several startups </li></ul><ul><li>Visual Studio team at Microsoft (DHTML, XML, Web Services) </li></ul><ul><li>3.5 Years with Amazon </li></ul><ul><li>Amazon Web Services Evangelist </li></ul>
    4. 4. What Is Amazon? <ul><li>Online Retailer </li></ul><ul><ul><li>Over 55 million active customer accounts </li></ul></ul><ul><ul><li>Seven countries: US, UK, Germany, Japan, France, Canada, China </li></ul></ul><ul><li>Technology Consumer </li></ul><ul><ul><li>Multi-National Web Sites </li></ul></ul><ul><ul><li>Vast Data Warehouse – 25 TB </li></ul></ul><ul><ul><li>World-Class Logistics – 21 fulfillment centers; 9 million ft2 </li></ul></ul><ul><li>Technology Provider </li></ul><ul><ul><li>Hundreds of thousands of Amazon Associates </li></ul></ul><ul><ul><li>Over 1,050,000 active seller accounts </li></ul></ul><ul><ul><li>Over 150,000 software developers registered to use Amazon Web Services </li></ul></ul>
    5. 5. Web Service Architecture Operating System Libraries / DLLs / Assemblies Application Internet Web Services Application
    6. 6. What Is Amazon Web Services? <ul><li>APIs that give developers programmatic access to Amazon’s data and technology </li></ul><ul><ul><li>Building-block web services </li></ul></ul><ul><ul><li>Web-scale infrastructure </li></ul></ul><ul><ul><li>E-commerce capability </li></ul></ul><ul><ul><li>Content, data, and information </li></ul></ul><ul><ul><li>New business models </li></ul></ul><ul><ul><li>Customer-created content </li></ul></ul>
    7. 7. AWS Product Family <ul><li>Amazon E-Commerce Service </li></ul><ul><ul><li>Complete access to Amazon’s product catalog </li></ul></ul><ul><ul><li>Free + Associates commissions paid </li></ul></ul><ul><li>Amazon Historical Pricing </li></ul><ul><ul><li>Data warehouse access for product pricing </li></ul></ul><ul><ul><li>Monthly Fee </li></ul></ul><ul><li>Amazon Mechanical Turk </li></ul><ul><ul><li>Artificial Artificial Intelligence </li></ul></ul><ul><ul><li>10% Commission </li></ul></ul><ul><ul><li>Paid workforce </li></ul></ul><ul><li>Amazon Simple Queue Service </li></ul><ul><ul><li>IT building block </li></ul></ul><ul><ul><li>In beta </li></ul></ul><ul><li>Amazon S3 </li></ul><ul><ul><li>Storage for the internet </li></ul></ul><ul><ul><li>Charge by storage/bandwidth usage </li></ul></ul><ul><li>Alexa Web Information Service </li></ul><ul><ul><li>Data warehouse access for web crawl data </li></ul></ul><ul><ul><li>10K calls per month free, then 15 cents per 1000 calls </li></ul></ul><ul><li>Alexa Web Search Platform </li></ul><ul><ul><li>Roll your own search engine </li></ul></ul><ul><ul><li>Pay for time, storage, bandwidth </li></ul></ul><ul><li>Alexa Top Sites </li></ul><ul><ul><li>Top sites by Alexa traffic rank </li></ul></ul><ul><ul><li>Charges by URL </li></ul></ul>
    8. 8. Amazon S3 Simple storage service <ul><li>Storage for the internet - web service to read and write data </li></ul><ul><li>15 cents per Gigabyte-Month to store data </li></ul><ul><li>20 cents per Gigabyte to access data </li></ul><ul><li>Private and public storage </li></ul><ul><li>Scalable, reliable, cost-effective, and simple! </li></ul>
    9. 9. Amazon Mechanical Turk
    10. 10. Man Versus Machine
    11. 11. But Not Really
    12. 12. Chair Or Table?
    13. 13. Survey Says…
    14. 14. Amazon Mechanical Turk <ul><li>Provides Web Services API for computers to integrate artificial artificial intelligence directly into their processing by making requests of humans </li></ul>
    15. 15. Put The Human In The Loop <ul><li>while (read (photo)) </li></ul><ul><li>{ </li></ul><ul><li> ContainsHuman = CallMechanicalTurk(photo); </li></ul><ul><li> if (ContainsHuman) </li></ul><ul><li>{     acceptPhoto(photo);   } else </li></ul><ul><li>{     rejectPhoto(photo); } </li></ul><ul><li>} </li></ul>
    16. 16. How It Works www.mturk.com Workers Artificial, Artificially Intelligent Software Requester (Developer) Human Intelligence Tasks (HITs) Completed HITs Worker Qualifications
    17. 17. What This Means <ul><li>For Software Developers </li></ul><ul><li>Solves problem of building applications that until now have not worked well without human intelligence </li></ul><ul><li>For Businesses </li></ul><ul><li>Efficiently, effectively and inexpensively complete millions of tasks that require human intelligence </li></ul><ul><li>For Anyone </li></ul><ul><li>Make money using the skills you already have whenever your schedule allows </li></ul><ul><li>Previously unimaginable business ideas are now feasible… </li></ul>
    18. 18. High Quality, Inexpensive, Massive Scale Image Clean-Up Which picture best represents this business?
    19. 19. <ul><li>Easy-to-use application to build HITs for Mturk </li></ul><ul><li>Requester-oriented interface: </li></ul><ul><ul><li>Create and issue a series of HITs (Human Intelligence Tasks) </li></ul></ul><ul><ul><li>There's no need to deal with source code or with raw XML data </li></ul></ul>Free Use of HIT-Builder Functions during Beta Test period (requires Amazon Requester account)
    20. 20. Building A HIT Hit- Builder.com
    21. 21. Building Applications With Amazon Mechanical Turk
    22. 22. API Status <ul><li>Requesters </li></ul><ul><ul><li>Must use APIs </li></ul></ul><ul><ul><li>Must have development skills </li></ul></ul><ul><ul><li>Expect some small tweaks during the beta </li></ul></ul><ul><li>Workers </li></ul><ul><ul><li>Must use mturk.com </li></ul></ul><ul><ul><li>Worker API in the future </li></ul></ul>
    23. 23. API Basics - #1 <ul><li>Use REST or SOAP, your choice </li></ul><ul><li>Be sure to check for IsValid in each response </li></ul><ul><li>URL encode Question and Answer XML data </li></ul><ul><li>Include proper namespace in XML data </li></ul>
    24. 24. API Basics - #2 <ul><li>Sign each request using your Secret Key: </li></ul>HMAC (Hashing Function) Service Operation Timestamp Secret Key Base 64 Encode SOAP Request REST Request URL Encode
    25. 25. System Design Process <ul><li>Design Great HITs </li></ul><ul><li>Identify Worker Qualifications </li></ul><ul><li>Determine approval strategy </li></ul><ul><li>Build Application </li></ul><ul><li>Deposit funds into Amazon account </li></ul><ul><li>Load Qualifications </li></ul><ul><li>Load HITs </li></ul><ul><li>Approve completed Assignments </li></ul><ul><li>Retrieve results </li></ul>
    26. 26. Designing A Good HIT - #1 <ul><li>Question elements </li></ul><ul><ul><li>Title </li></ul></ul><ul><ul><li>Text Block </li></ul></ul><ul><ul><li>List </li></ul></ul><ul><ul><li>Binary </li></ul></ul><ul><li>Answer elements </li></ul><ul><ul><li>Free Text </li></ul></ul><ul><ul><li>Selection (1 or multiple) </li></ul></ul><ul><li>Build XML using the Question Builder </li></ul>
    27. 27. Designing A Good HIT - #2 <ul><li>HIT design considerations </li></ul><ul><ul><li>Be clear </li></ul></ul><ul><ul><li>Be verbose, if necessary </li></ul></ul><ul><ul><li>Be specific about needs & expectations </li></ul></ul><ul><ul><li>Leave no ambiguities </li></ul></ul><ul><ul><li>Test drive and monitor before going full-bore </li></ul></ul><ul><ul><li>Always have an appropriate “none of the above” choice </li></ul></ul><ul><ul><li>Punctuation, capitalization, etc. </li></ul></ul><ul><ul><li>Approval strategy </li></ul></ul>
    28. 28. Identify Worker Qualifications <ul><li>Create a test to admit only workers with the skills or other qualifications that are needed for your particular type of HITs </li></ul><ul><li>Qualification often looks very much like a HIT </li></ul><ul><li>Objective (multiple choice) or subjective (free text) </li></ul><ul><li>If necessary, use “Secret Code” qualifications </li></ul><ul><li>Use or build on existing Qualifications </li></ul>
    29. 29. Application Flow Load Qualifications Load HITs Any Completed Assignments? Retrieve and Approve Do Local Processing More Work? Y Y Done! Do Local Processing N
    30. 30. Load Qualifications <ul><li>Relevant API Calls </li></ul><ul><ul><li>CreateQualificationType </li></ul></ul><ul><ul><li>GetQualificationRequests </li></ul></ul><ul><ul><li>GrantQualification </li></ul></ul><ul><li>Notes </li></ul><ul><ul><li>Qualifications which include an Answer Key are automatically granted when passed </li></ul></ul><ul><ul><li>Application must explicitly check and grant others using GrantQualification </li></ul></ul><ul><ul><li>You cannot currently attach a Qualification to a worker without a prior request </li></ul></ul>
    31. 31. Load HITs <ul><li>Relevant API Calls </li></ul><ul><ul><li>CreateHIT - Returns HIT identifier </li></ul></ul><ul><ul><li>DisableHIT – Turn it off </li></ul></ul><ul><ul><li>ExtendHIT – Add additional time </li></ul></ul><ul><li>Notes </li></ul><ul><ul><li>Multiple assignments per HIT are important </li></ul></ul><ul><ul><li>Auto approval time </li></ul></ul><ul><ul><li>Qualification requirements can be attached to HITs </li></ul></ul><ul><ul><li>Attach application data using RequesterAnnotation </li></ul></ul>
    32. 32. Poll And Approve Work <ul><li>Relevant API Calls </li></ul><ul><ul><li>GetReviewableHITs </li></ul></ul><ul><ul><li>GetHITs </li></ul></ul><ul><ul><li>GetAssignmentsForHIT </li></ul></ul><ul><ul><li>ApproveAssignment – Pay the Worker </li></ul></ul><ul><ul><li>RejectAssignment – Don’t pay the Worker </li></ul></ul><ul><ul><li>DisposeHIT </li></ul></ul><ul><ul><li>ExtendHIT </li></ul></ul><ul><ul><li>NotifyWorkers – Send e-mail notification </li></ul></ul>
    33. 33. Retrieve Results <ul><li>Relevant API calls </li></ul><ul><ul><li>GetAssignmentsForHIT – Get the actual answers </li></ul></ul><ul><li>Notes </li></ul><ul><ul><li>GetReviewableHITs returns list of non-assignable HITs </li></ul></ul><ul><ul><li>Dispose or Extend (or they’ll keep coming back) </li></ul></ul>
    34. 34. Perform Local Processing <ul><li>Specific to your application </li></ul><ul><li>Types </li></ul><ul><ul><li>Incremental (per Assignment) </li></ul></ul><ul><ul><li>Batch (per group of related HITs) </li></ul></ul><ul><li>Connect application logic back to Turk </li></ul><ul><ul><li>If application’s needs change, Dispose of Associated HIT </li></ul></ul><ul><ul><li>Adjust pricing and timing dynamically </li></ul></ul><ul><ul><li>Adjust necessary Qualifications </li></ul></ul>
    35. 35. Next Steps… <ul><li>Find potential HITs: </li></ul><ul><ul><li>Within your organization </li></ul></ul><ul><ul><li>Unmet or underserved market </li></ul></ul><ul><li>Design workflow process around the Mechanical Turk </li></ul><ul><li>Start writing code and loading HITs </li></ul><ul><li>Process results </li></ul><ul><li>Get your work done! </li></ul>
    36. 36. Q & A
    37. 37. Thanks!
    38. 38. For More Information: Mechanical Turk: www.mturk.com AWS Blog: aws.typepad.com Amazon Web Services: aws.amazon.com © 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
    39. 39. © 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×