Artificial Artificial Intelligence: Using Amazon Mechanical Turk and .NET to Create a…


Published on

Amazon Mechanical Turk is a new Web service that allows .NET software developers to incorporate the power of human decision-making into their automated software systems

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Artificial Artificial Intelligence: Using Amazon Mechanical Turk and .NET to Create a…

    1. 1. Artificial, Artificial Intelligence Using Amazon Mechanical Turk and .NET to Create a New Breed of Web App Jeff Barr Web Services Evangelist Amazon Web Services NGW012
    2. 2. Agenda <ul><li>Amazon Web Services Overview </li></ul><ul><li>Artificial, Artificial Intelligence </li></ul><ul><li>Demos </li></ul><ul><li>Writing Mechanical Turk Applications </li></ul><ul><li>Q&A </li></ul>
    3. 3. Introduction And Background <ul><li>Software development background </li></ul><ul><li>Veteran of several startups </li></ul><ul><li>Visual Studio team at Microsoft (DHTML, XML, Web Services) </li></ul><ul><li>3.5 Years with Amazon </li></ul><ul><li>Amazon Web Services Evangelist </li></ul>
    4. 4. What Is Amazon? <ul><li>Online Retailer </li></ul><ul><ul><li>Over 55 million active customer accounts </li></ul></ul><ul><ul><li>Seven countries: US, UK, Germany, Japan, France, Canada, China </li></ul></ul><ul><li>Technology Consumer </li></ul><ul><ul><li>Multi-National Web Sites </li></ul></ul><ul><ul><li>Vast Data Warehouse – 25 TB </li></ul></ul><ul><ul><li>World-Class Logistics – 21 fulfillment centers; 9 million ft2 </li></ul></ul><ul><li>Technology Provider </li></ul><ul><ul><li>Hundreds of thousands of Amazon Associates </li></ul></ul><ul><ul><li>Over 1,050,000 active seller accounts </li></ul></ul><ul><ul><li>Over 150,000 software developers registered to use Amazon Web Services </li></ul></ul>
    5. 5. Web Service Architecture Operating System Libraries / DLLs / Assemblies Application Internet Web Services Application
    6. 6. What Is Amazon Web Services? <ul><li>APIs that give developers programmatic access to Amazon’s data and technology </li></ul><ul><ul><li>Building-block web services </li></ul></ul><ul><ul><li>Web-scale infrastructure </li></ul></ul><ul><ul><li>E-commerce capability </li></ul></ul><ul><ul><li>Content, data, and information </li></ul></ul><ul><ul><li>New business models </li></ul></ul><ul><ul><li>Customer-created content </li></ul></ul>
    7. 7. AWS Product Family <ul><li>Amazon E-Commerce Service </li></ul><ul><ul><li>Complete access to Amazon’s product catalog </li></ul></ul><ul><ul><li>Free + Associates commissions paid </li></ul></ul><ul><li>Amazon Historical Pricing </li></ul><ul><ul><li>Data warehouse access for product pricing </li></ul></ul><ul><ul><li>Monthly Fee </li></ul></ul><ul><li>Amazon Mechanical Turk </li></ul><ul><ul><li>Artificial Artificial Intelligence </li></ul></ul><ul><ul><li>10% Commission </li></ul></ul><ul><ul><li>Paid workforce </li></ul></ul><ul><li>Amazon Simple Queue Service </li></ul><ul><ul><li>IT building block </li></ul></ul><ul><ul><li>In beta </li></ul></ul><ul><li>Amazon S3 </li></ul><ul><ul><li>Storage for the internet </li></ul></ul><ul><ul><li>Charge by storage/bandwidth usage </li></ul></ul><ul><li>Alexa Web Information Service </li></ul><ul><ul><li>Data warehouse access for web crawl data </li></ul></ul><ul><ul><li>10K calls per month free, then 15 cents per 1000 calls </li></ul></ul><ul><li>Alexa Web Search Platform </li></ul><ul><ul><li>Roll your own search engine </li></ul></ul><ul><ul><li>Pay for time, storage, bandwidth </li></ul></ul><ul><li>Alexa Top Sites </li></ul><ul><ul><li>Top sites by Alexa traffic rank </li></ul></ul><ul><ul><li>Charges by URL </li></ul></ul>
    8. 8. Amazon S3 Simple storage service <ul><li>Storage for the internet - web service to read and write data </li></ul><ul><li>15 cents per Gigabyte-Month to store data </li></ul><ul><li>20 cents per Gigabyte to access data </li></ul><ul><li>Private and public storage </li></ul><ul><li>Scalable, reliable, cost-effective, and simple! </li></ul>
    9. 9. Amazon Mechanical Turk
    10. 10. Man Versus Machine
    11. 11. But Not Really
    12. 12. Chair Or Table?
    13. 13. Survey Says…
    14. 14. Amazon Mechanical Turk <ul><li>Provides Web Services API for computers to integrate artificial artificial intelligence directly into their processing by making requests of humans </li></ul>
    15. 15. Put The Human In The Loop <ul><li>while (read (photo)) </li></ul><ul><li>{ </li></ul><ul><li> ContainsHuman = CallMechanicalTurk(photo); </li></ul><ul><li> if (ContainsHuman) </li></ul><ul><li>{     acceptPhoto(photo);   } else </li></ul><ul><li>{     rejectPhoto(photo); } </li></ul><ul><li>} </li></ul>
    16. 16. How It Works Workers Artificial, Artificially Intelligent Software Requester (Developer) Human Intelligence Tasks (HITs) Completed HITs Worker Qualifications
    17. 17. What This Means <ul><li>For Software Developers </li></ul><ul><li>Solves problem of building applications that until now have not worked well without human intelligence </li></ul><ul><li>For Businesses </li></ul><ul><li>Efficiently, effectively and inexpensively complete millions of tasks that require human intelligence </li></ul><ul><li>For Anyone </li></ul><ul><li>Make money using the skills you already have whenever your schedule allows </li></ul><ul><li>Previously unimaginable business ideas are now feasible… </li></ul>
    18. 18. High Quality, Inexpensive, Massive Scale Image Clean-Up Which picture best represents this business?
    19. 19. <ul><li>Easy-to-use application to build HITs for Mturk </li></ul><ul><li>Requester-oriented interface: </li></ul><ul><ul><li>Create and issue a series of HITs (Human Intelligence Tasks) </li></ul></ul><ul><ul><li>There's no need to deal with source code or with raw XML data </li></ul></ul>Free Use of HIT-Builder Functions during Beta Test period (requires Amazon Requester account)
    20. 20. Building A HIT Hit-
    21. 21. Building Applications With Amazon Mechanical Turk
    22. 22. API Status <ul><li>Requesters </li></ul><ul><ul><li>Must use APIs </li></ul></ul><ul><ul><li>Must have development skills </li></ul></ul><ul><ul><li>Expect some small tweaks during the beta </li></ul></ul><ul><li>Workers </li></ul><ul><ul><li>Must use </li></ul></ul><ul><ul><li>Worker API in the future </li></ul></ul>
    23. 23. API Basics - #1 <ul><li>Use REST or SOAP, your choice </li></ul><ul><li>Be sure to check for IsValid in each response </li></ul><ul><li>URL encode Question and Answer XML data </li></ul><ul><li>Include proper namespace in XML data </li></ul>
    24. 24. API Basics - #2 <ul><li>Sign each request using your Secret Key: </li></ul>HMAC (Hashing Function) Service Operation Timestamp Secret Key Base 64 Encode SOAP Request REST Request URL Encode
    25. 25. System Design Process <ul><li>Design Great HITs </li></ul><ul><li>Identify Worker Qualifications </li></ul><ul><li>Determine approval strategy </li></ul><ul><li>Build Application </li></ul><ul><li>Deposit funds into Amazon account </li></ul><ul><li>Load Qualifications </li></ul><ul><li>Load HITs </li></ul><ul><li>Approve completed Assignments </li></ul><ul><li>Retrieve results </li></ul>
    26. 26. Designing A Good HIT - #1 <ul><li>Question elements </li></ul><ul><ul><li>Title </li></ul></ul><ul><ul><li>Text Block </li></ul></ul><ul><ul><li>List </li></ul></ul><ul><ul><li>Binary </li></ul></ul><ul><li>Answer elements </li></ul><ul><ul><li>Free Text </li></ul></ul><ul><ul><li>Selection (1 or multiple) </li></ul></ul><ul><li>Build XML using the Question Builder </li></ul>
    27. 27. Designing A Good HIT - #2 <ul><li>HIT design considerations </li></ul><ul><ul><li>Be clear </li></ul></ul><ul><ul><li>Be verbose, if necessary </li></ul></ul><ul><ul><li>Be specific about needs & expectations </li></ul></ul><ul><ul><li>Leave no ambiguities </li></ul></ul><ul><ul><li>Test drive and monitor before going full-bore </li></ul></ul><ul><ul><li>Always have an appropriate “none of the above” choice </li></ul></ul><ul><ul><li>Punctuation, capitalization, etc. </li></ul></ul><ul><ul><li>Approval strategy </li></ul></ul>
    28. 28. Identify Worker Qualifications <ul><li>Create a test to admit only workers with the skills or other qualifications that are needed for your particular type of HITs </li></ul><ul><li>Qualification often looks very much like a HIT </li></ul><ul><li>Objective (multiple choice) or subjective (free text) </li></ul><ul><li>If necessary, use “Secret Code” qualifications </li></ul><ul><li>Use or build on existing Qualifications </li></ul>
    29. 29. Application Flow Load Qualifications Load HITs Any Completed Assignments? Retrieve and Approve Do Local Processing More Work? Y Y Done! Do Local Processing N
    30. 30. Load Qualifications <ul><li>Relevant API Calls </li></ul><ul><ul><li>CreateQualificationType </li></ul></ul><ul><ul><li>GetQualificationRequests </li></ul></ul><ul><ul><li>GrantQualification </li></ul></ul><ul><li>Notes </li></ul><ul><ul><li>Qualifications which include an Answer Key are automatically granted when passed </li></ul></ul><ul><ul><li>Application must explicitly check and grant others using GrantQualification </li></ul></ul><ul><ul><li>You cannot currently attach a Qualification to a worker without a prior request </li></ul></ul>
    31. 31. Load HITs <ul><li>Relevant API Calls </li></ul><ul><ul><li>CreateHIT - Returns HIT identifier </li></ul></ul><ul><ul><li>DisableHIT – Turn it off </li></ul></ul><ul><ul><li>ExtendHIT – Add additional time </li></ul></ul><ul><li>Notes </li></ul><ul><ul><li>Multiple assignments per HIT are important </li></ul></ul><ul><ul><li>Auto approval time </li></ul></ul><ul><ul><li>Qualification requirements can be attached to HITs </li></ul></ul><ul><ul><li>Attach application data using RequesterAnnotation </li></ul></ul>
    32. 32. Poll And Approve Work <ul><li>Relevant API Calls </li></ul><ul><ul><li>GetReviewableHITs </li></ul></ul><ul><ul><li>GetHITs </li></ul></ul><ul><ul><li>GetAssignmentsForHIT </li></ul></ul><ul><ul><li>ApproveAssignment – Pay the Worker </li></ul></ul><ul><ul><li>RejectAssignment – Don’t pay the Worker </li></ul></ul><ul><ul><li>DisposeHIT </li></ul></ul><ul><ul><li>ExtendHIT </li></ul></ul><ul><ul><li>NotifyWorkers – Send e-mail notification </li></ul></ul>
    33. 33. Retrieve Results <ul><li>Relevant API calls </li></ul><ul><ul><li>GetAssignmentsForHIT – Get the actual answers </li></ul></ul><ul><li>Notes </li></ul><ul><ul><li>GetReviewableHITs returns list of non-assignable HITs </li></ul></ul><ul><ul><li>Dispose or Extend (or they’ll keep coming back) </li></ul></ul>
    34. 34. Perform Local Processing <ul><li>Specific to your application </li></ul><ul><li>Types </li></ul><ul><ul><li>Incremental (per Assignment) </li></ul></ul><ul><ul><li>Batch (per group of related HITs) </li></ul></ul><ul><li>Connect application logic back to Turk </li></ul><ul><ul><li>If application’s needs change, Dispose of Associated HIT </li></ul></ul><ul><ul><li>Adjust pricing and timing dynamically </li></ul></ul><ul><ul><li>Adjust necessary Qualifications </li></ul></ul>
    35. 35. Next Steps… <ul><li>Find potential HITs: </li></ul><ul><ul><li>Within your organization </li></ul></ul><ul><ul><li>Unmet or underserved market </li></ul></ul><ul><li>Design workflow process around the Mechanical Turk </li></ul><ul><li>Start writing code and loading HITs </li></ul><ul><li>Process results </li></ul><ul><li>Get your work done! </li></ul>
    36. 36. Q & A
    37. 37. Thanks!
    38. 38. For More Information: Mechanical Turk: AWS Blog: Amazon Web Services: © 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
    39. 39. © 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.