Best Practices for Sentiment Analysis Webinar


Published on

An informative session on Amazon Mechanical Turk where you will learn how your company can leverage the human crowd for human sentiment analysis of content such as tweets, articles, RSS feeds and blog posts. This session digs into the details of getting started and provides information on how to be successful so you get accurate results. Additionally, FreedomOSS will share their experiences designing and managing sentiment tasks and demo's their CrowdControl crowdsourcing platform that is built on top of Mechanical Turk.

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Hi, I want to welcome you all to the Webinar on how to apply Mechanical turk to a sentiment analysis solution. So, allow me to begin by introducing today’s speakers. My name is john Hoskins, I am the Senior Manager of Business Development for the Mechancial Turk team here at Amazon Web services. With me is Max Yankelevich, Chief Architect of the CrowdControl solution at FreedomOSS – a Mechanical Turk Solution Provider Partner.
  • First just some logistics. At the end of the webinar we’ll be answering questions – to send us a question, please enter it in to the question panel on the GoToMeeting control panel. We’ll answer as many questions as possible. Today we’re going to briefly introduce sentiment analysis – how to derive it, how can you can apply human judgment on a scalable basis. Finally, we’ll introduce FreedomOSS, one of our partners that has experience at implementing large scale sentiment analysis solution using their CrowdControl platform with Mechanical Turk.
  • How could you use Sentiment analysis? What advantage does sit bring to you? With the proliferation of reviews, ratings, recommendations and other forms of online expression, online opinion has turned into a kind of virtual currency for businesses looking to market their products, identify new opportunities and manage their reputations.
  • Social media is dominated by people who often express complex emotions such as sarcasm, especially in Tweets and blog posts. Most platforms can't deal with wit, sarcasm and complex emotions which make them unsuitable for many of the questions Many “negative" statements online turn out to be positive while many social mentions consider to be positive turn out to be negative when a human examines them SPAM – relevance cleansing dirty data – not to mention the complexities of analyzing “text language” – the ever changing short cuts expressing
  • So, to maintain the fast pace of posts online – is it possible to use human judgement?
  • Amazon Mechanical Turk It is a marketplace for work. In a traditional marketplace, like the retail site that you’re probably already familiar with, there are buyers who spend money to purchase products and there are sellers who make money by selling their products. This is true in the Amazon Mechanical Turk Marketplace as well. Businesses, or Requesters as they are called on Mechanical Turk are buying work. Mechanical Turk Workers sell their work and earn money when the transaction is complete. And just like in the retail experience, Amazon Mechanical Turk takes care of all of the transaction details including getting the work “listed” on the site so Workers see it when they browse for Work as well as completing the payment transaction when work is approved by you, the Requester. Let’s talk in a bit more detail about how it works.
  • Requesters design HITs, Human intelligence tasks. These HITs are composed of Assignments which are published to the Mechanical Turk Marketplace. Workers select tasks to complete and submit their assignments to Requesters. Requesters approve or reject the results and pay Workers.
  • How can I automatically decide if I have the best answer? Plurality – ask more than one worker and compare responses. Train the worker population – pass competencies, show competencies, maintain competencies. Ongoing tests – gold standards, work elements for which you know the answer.
  • At this point I am going to turn over the presentation to Max Yankelevich who will discuss how FreedomOSS implemented a sentiment analysis workflow using their platform CrowdControl and Mechanical turk.
  • For more information, please feel free to reach out to Max and myself directly at these emails. We’ll be posting the webinar on our site and will send out a notification as soon as it’s available.
  • Best Practices for Sentiment Analysis Webinar

    1. 1. Best Practices for Sentiment Analysis Presented by: John Hoskins Amazon Mechanical Turk Max Yankelevich Freedom OSS CrowdControl
    2. 2. Introductions
    3. 3. Welcome <ul><li>Logistics </li></ul><ul><ul><li>How to ask a question:. </li></ul></ul><ul><ul><ul><li>Ask a question with the question panel. </li></ul></ul></ul><ul><ul><ul><li>We will moderate questions. </li></ul></ul></ul><ul><li>Agenda </li></ul><ul><ul><li>Introductions </li></ul></ul><ul><ul><li>Sentiment Analysis </li></ul></ul><ul><ul><li>Applying Mechanical Turk </li></ul></ul><ul><ul><li>CrowdControl – optimized for Sentiment </li></ul></ul><ul><ul><li>Q & A </li></ul></ul>
    4. 4. What is Sentiment Analysis? <ul><li>Keeping your finger on the pulse of your market – in near real time With the explosion in use of Facebook, Twitter, and blogging – it is essential to hear the true Voice of the Customer. Yet keeping up with the round-the-clock information torrent is impossible with old, manual methods. As businesses look to automate the process of filtering out the noise, understanding the conversations, identifying the relevant content and actioning it appropriately, many are now looking to the field of sentiment analysis [ </li></ul>
    5. 5. Why Sentiment Analysis? <ul><li>The ability to lead the competition </li></ul><ul><ul><ul><li>in customer satisfaction and support, </li></ul></ul></ul><ul><ul><ul><li>brand and reputation management </li></ul></ul></ul><ul><ul><ul><li>product design and marketing </li></ul></ul></ul><ul><ul><ul><li>With the proliferation of reviews, ratings, recommendations and other forms of online expression, online opinion has turned into a kind of virtual currency for businesses looking to market their products, identify new opportunities and manage their reputations. </li></ul></ul></ul>
    6. 6. How to properly analyze Sentiment.
    7. 7. Context & Cognitive Recognition are the keys to an accurate analysis <ul><li>Human analysis provides more accurate assessment. </li></ul><ul><li>Complex emotions such as sarcasm are presented in tweets and blog posts. </li></ul><ul><li>Wit, sarcasm and complex emotions are difficult to analyze with technology alone </li></ul><ul><li>Processing the unnatural language of text messaging: lol, omg </li></ul><ul><li>Positive and negative emotions are difficult to accurately assess </li></ul><ul><li>Avoid misleading classification of SPAM </li></ul>
    8. 8. Is human judgment affordable and scalable? <ul><li>Yes. </li></ul>
    9. 9. Introducing Mechanical Turk <ul><li>An affordable solution for human judgment. </li></ul>
    10. 10. Mechanical Turk is a marketplace for work . <ul><li>Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce. </li></ul><ul><ul><li>Flexibility : Scale your workforce up and down quickly </li></ul></ul><ul><ul><li>Accuracy : Get high quality, efficient and cost effective results. </li></ul></ul><ul><ul><li>Price : Pay only when you are satisfied with the results. </li></ul></ul><ul><ul><li>Speed : Start receiving results in minutes </li></ul></ul>
    11. 11. Workforce <ul><li>500,000 Workers </li></ul><ul><li>190+ Countries </li></ul><ul><li>Who are the Workers? </li></ul><ul><ul><li>Workers are global: 24X7 Follow the Sun </li></ul></ul><ul><li>Managing Your Workforce </li></ul><ul><ul><li>Can narrow to US based (i.e. when you need western culture competence) </li></ul></ul><ul><ul><li>Leverage qualifications to find your best Workers </li></ul></ul>
    12. 12. How it Works
    13. 13. Popular Use Cases <ul><li>Data Management </li></ul><ul><ul><li>Data Verification </li></ul></ul><ul><ul><li>Data Entry & Collection </li></ul></ul><ul><ul><li>Data De-duplication </li></ul></ul><ul><ul><li>Algorithm Training </li></ul></ul><ul><li>Content & Media </li></ul><ul><ul><li>Moderate Photos & Content </li></ul></ul><ul><ul><li>Content Creation & Editing </li></ul></ul><ul><ul><li>Transcription </li></ul></ul><ul><li>Categorization </li></ul><ul><ul><li>Classification </li></ul></ul><ul><ul><li>Tagging </li></ul></ul><ul><ul><li>Sentiment Analysis </li></ul></ul><ul><li>Business Services </li></ul><ul><ul><li>Search Relevancy </li></ul></ul><ul><ul><li>Product Usability Testing </li></ul></ul><ul><ul><li>Research </li></ul></ul>
    14. 14. Case Study – Sentiment Analysis <ul><li>Problem: A large consumer brand reporting customer need 10’s of thousands of human coded samples to establish a baseline – for each project </li></ul><ul><li>Solution: Code them through Mechanical Turk. </li></ul><ul><li>Details: </li></ul><ul><ul><li>Projects are completed in hours providing faster to market production of broad analysis. </li></ul></ul><ul><ul><li>By integrating Mechanical Turk into their process, they freed up analysts time to do value add work and can start projects in a day instead of weeks.  </li></ul></ul>
    15. 15. Strategies for Analyzing Results <ul><li>Plurality Asking multiple Workers to do assignments (if their answers agree then the result is validated) </li></ul><ul><li>Qualifying & Training Workers Assess competence on your coding instructions before allowing them to work for you </li></ul><ul><li>Known Data Sets Include work with known answers to quickly assess worker accuracy. </li></ul>
    16. 16. Implementing a Sentiment Analysis Workflow
    17. 17. Freedom at a Glance <ul><li>Long term AWS Partner </li></ul><ul><ul><li>Enterprise Cloud Business </li></ul></ul><ul><ul><li>CrowdControl – fastest growing division </li></ul></ul><ul><li>Corporate Office: Newtown PA </li></ul><ul><ul><li>Regional Offices: Newark, NJ. Reston, VA, Seattle, WA </li></ul></ul><ul><ul><li>Offshore Engineering Centers </li></ul></ul><ul><ul><li>Global operations centers </li></ul></ul><ul><li>Privately held </li></ul><ul><ul><li>Established in 2008 </li></ul></ul><ul><ul><li>300% annual growth </li></ul></ul>
    18. 18. Workflow, Adjudication, Worker Management . <ul><li>Programmatic connection to crowdsourcing. </li></ul><ul><li>How do I know I have the best answer? </li></ul><ul><li>How do I break my workflow into discrete tasks </li></ul><ul><li>Who are my best workers? </li></ul><ul><li>How do I put it all together? </li></ul>
    19. 19. CrowdControl for Sentiment Analysis <ul><li>Provides High Sentiment Quality Data at Lower Cost </li></ul><ul><li>Ability to Process Large Number of UGC </li></ul><ul><li>Combines best of breed Artificial Intelligence to handle most challenging nuances of “Crowdsourcing” for efficient Sentiment Analysis </li></ul><ul><ul><li>Worker Management </li></ul></ul><ul><ul><li>Adjudication strategies </li></ul></ul><ul><ul><li>Workflows </li></ul></ul><ul><ul><li>Can easily retrieve and send data to and from any data source (e.g. Database, File ,etc.) </li></ul></ul>
    20. 20. What Can CrowdControl™ Do for Me? Manual Steps Mechanical Turk Portal Turkers Mechanical Turk Portal Turkers Customer IT Systems Customer IT Systems Complete Automation Quality Information
    21. 21. Sentiment Analysis Process Setup
    22. 22. Brand Sentiment Coding Template
    23. 24. John Hoskins, Amazon Web Services: [email_address] Max Yankelevich, FreedomOSS: [email_address] @amazonmturk
    24. 25. Q & A <ul><li>Your Questions. Answered. </li></ul>