Twitter has brought much attention recently as a hot research topic in the domain of sentiment analysis. Training sentiment classifiers from tweets data often faces the data sparsity problem partly due to the large variety of short and irregular forms introduced to tweets because of the 140-character limit. In this work we propose using two different sets of features to alleviate the data sparseness problem. One is the semantic feature set where we extract semantically hidden concepts from tweets and then incorporate them into classifier training through interpolation. Another is the sentiment-topic feature set where we extract latent topics and the associated topic sentiment from tweets, then augment the original feature space with these sentiment-topics. Experimental results on the Stanford Twitter Sentiment Dataset show that both feature sets outperform the baseline model using unigrams only. Moreover, using semantic features rivals the previously reported best result. Using sentiment-topic features achieves 86.3% sentiment classification accuracy, which outperforms existing approaches.
In this talk we cover
1. Why NLP and DL
2. Practical Challenges
3. Some Popular Deep Learning models for NLP
Today you can take any webpage in any language and translate it automatically into language you know! You can also cut paste an article or other document into NLP systems and immediately get list of companies and people it talks about, topics that are relevant and the sentiment of the document. When you talk to Google or Amazon assistant, you are using NLP systems. NLP is not perfect but given the advances in last two years and continuing, it is a growing field. Let’s see how it actually works, specifically using Deep learning
About Shishir
Shishir is a Senior Data Scientist at Thomson Reuters working on Deep Learning and NLP to solve real customer pain, even ones they have become used to.
A brief survey presentation about Arabic Question Answering touching the different Natural Language Processing and Information Retrieval Approaches to Question Analysis, Passage Retrieval and Answer Extraction. In addition to the listing of the different NLP tools used in AQA and the Challenges and future trends in this area.
Please if you want to cite this paper you can download it here:
http://www.acit2k.org/ACIT/2012Proceedings/13106.pdf
Sentiment Analysis also known as opinion mining and Emotional AI
Refers to the use of natural language processing, text analysis, computational linguistics and biometrics to systematically identify, extract, quantify and study affective states and subjective information.
widely used in
Reviews
Survey responses
Online and social media
Health care
Twitter has brought much attention recently as a hot research topic in the domain of sentiment analysis. Training sentiment classifiers from tweets data often faces the data sparsity problem partly due to the large variety of short and irregular forms introduced to tweets because of the 140-character limit. In this work we propose using two different sets of features to alleviate the data sparseness problem. One is the semantic feature set where we extract semantically hidden concepts from tweets and then incorporate them into classifier training through interpolation. Another is the sentiment-topic feature set where we extract latent topics and the associated topic sentiment from tweets, then augment the original feature space with these sentiment-topics. Experimental results on the Stanford Twitter Sentiment Dataset show that both feature sets outperform the baseline model using unigrams only. Moreover, using semantic features rivals the previously reported best result. Using sentiment-topic features achieves 86.3% sentiment classification accuracy, which outperforms existing approaches.
In this talk we cover
1. Why NLP and DL
2. Practical Challenges
3. Some Popular Deep Learning models for NLP
Today you can take any webpage in any language and translate it automatically into language you know! You can also cut paste an article or other document into NLP systems and immediately get list of companies and people it talks about, topics that are relevant and the sentiment of the document. When you talk to Google or Amazon assistant, you are using NLP systems. NLP is not perfect but given the advances in last two years and continuing, it is a growing field. Let’s see how it actually works, specifically using Deep learning
About Shishir
Shishir is a Senior Data Scientist at Thomson Reuters working on Deep Learning and NLP to solve real customer pain, even ones they have become used to.
A brief survey presentation about Arabic Question Answering touching the different Natural Language Processing and Information Retrieval Approaches to Question Analysis, Passage Retrieval and Answer Extraction. In addition to the listing of the different NLP tools used in AQA and the Challenges and future trends in this area.
Please if you want to cite this paper you can download it here:
http://www.acit2k.org/ACIT/2012Proceedings/13106.pdf
Sentiment Analysis also known as opinion mining and Emotional AI
Refers to the use of natural language processing, text analysis, computational linguistics and biometrics to systematically identify, extract, quantify and study affective states and subjective information.
widely used in
Reviews
Survey responses
Online and social media
Health care
NLP with Deep Learning Guest Lecture slides by Fatih Mehmet Güler, PragmaCraft. Includes my background on the subject, our projects, the NLP stages and the latest developments.
Question Answering System using machine learning approachGarima Nanda
In a compact form, this is a presentation reflecting how the machine learning approach can be used for the effective and efficient interaction using classification techniques.
Arabic is the 6th most wide-spread natural language in the world with more than 350 million native speakers. Arabic question answering systems are gaining great significance due to the increasing amounts of Arabic unstructured content on the Internet and the increasing demand for information that regular information retrieval techniques do not satisfy. Question answering systems generally, and Arabic systems are no exception, hit an upper bound of performance due to the propagation of error in their pipeline. This increases the significance of answer selection and validation systems as they enhance the certainty and accuracy of question answering systems. Very few works tackled the Arabic answer selection and validation problem, and they used the same question answering pipeline without any changes to satisfy the requirements of answer selection and validation. That is why they did not perform adequately well in this task. In this dissertation, a new approach to Arabic answer selection and validation is presented through “ALQASIM”, which is a QA4MRE (Question Answering for Machine Reading Evaluation) system. ALQASIM analyzes the reading test documents instead of the questions, utilizes sentence splitting, root expansion, and semantic expansion using an ontology built from the CLEF 2012 background collections. Our experiments have been conducted on the test-set provided by CLEF 2012 through the task of QA4MRE. This approach led to a promising performance of 0.36 Accuracy and 0.42 C@1, which is double the performance of the best performing Arabic QA4MRE system.
Publications:
http://scholar.google.com/citations?user=XGJiEioAAAAJ&hl=en
https://aast.academia.edu/AhmedMagdy
Sentiment analysis and opinion mining is almost same thing however there is minor difference between them that is opinion mining extracts and analyze people's opinion about an entity while Sentiment analysis search for the sentiment words/expression in a text and then analyze it.
It uses machine learning techniques like SVM (Support Vector Machines) to analyze the text and classify them as positive, negative or neutral.
Sentiment analysis involves the process of automatically detecting the polarity of a text and extracting the author's reviews on the subject, and finally, classifying the text. In many research approaches, the textual data classification is done using deep learning models. This is due to the ability of deep learning models to classify a text with a high accuracy and the ability to model the sequence of textual data with word dependencies throughout the sentence. One of these deep learning models is RNN (Recurrent Neural Network). In order to use these models, the textual data and words must be converted into numerical vectors, for which various algorithms and methods have been proposed [10]. Today's pretrained word embedding libraries such as FastText have a high accuracy and quality in vector representations for words. Accordingly, in most current systems and research approaches, these libraries are used to convert the textual data to numerical vectors
II-SDV 2017: Localizing International Content for Search, Data Mining and Ana...Dr. Haxel Consult
Advances in text mining, analytics and machine learning are transforming our applications and enabling ever more powerful applications, yet most applications and platforms are designed to deal with a single (normalized) language. Hence as our applications and platforms are increasingly required to ingest international content, the challenge becomes to find ways to normalize content to a single language without compromising quality. An extension of this question in terms of such applications is also how we define quality in this context and what, if any, bi-products a localization effort can produce that may enhance the usefulness of the application.
This talk will, using patent searching as an example use case, review the challenges and possible solution approaches for handling localization effectively and will show what current emerging technology offers, what to expect and what not to expect and provide an introductory practical guide to handling localization in the context of data mining and analytics.
Enhancing Enterprise Search with Machine Learning - Simon Hughes, Dice.comSimon Hughes
In the talk I describe two approaches for improve the recall and precision of an enterprise search engine using machine learning techniques. The main focus is improving relevancy with ML while using your existing search stack, be that Luce, Solr, Elastic Search, Endeca or something else.
NLP with Deep Learning Guest Lecture slides by Fatih Mehmet Güler, PragmaCraft. Includes my background on the subject, our projects, the NLP stages and the latest developments.
Question Answering System using machine learning approachGarima Nanda
In a compact form, this is a presentation reflecting how the machine learning approach can be used for the effective and efficient interaction using classification techniques.
Arabic is the 6th most wide-spread natural language in the world with more than 350 million native speakers. Arabic question answering systems are gaining great significance due to the increasing amounts of Arabic unstructured content on the Internet and the increasing demand for information that regular information retrieval techniques do not satisfy. Question answering systems generally, and Arabic systems are no exception, hit an upper bound of performance due to the propagation of error in their pipeline. This increases the significance of answer selection and validation systems as they enhance the certainty and accuracy of question answering systems. Very few works tackled the Arabic answer selection and validation problem, and they used the same question answering pipeline without any changes to satisfy the requirements of answer selection and validation. That is why they did not perform adequately well in this task. In this dissertation, a new approach to Arabic answer selection and validation is presented through “ALQASIM”, which is a QA4MRE (Question Answering for Machine Reading Evaluation) system. ALQASIM analyzes the reading test documents instead of the questions, utilizes sentence splitting, root expansion, and semantic expansion using an ontology built from the CLEF 2012 background collections. Our experiments have been conducted on the test-set provided by CLEF 2012 through the task of QA4MRE. This approach led to a promising performance of 0.36 Accuracy and 0.42 C@1, which is double the performance of the best performing Arabic QA4MRE system.
Publications:
http://scholar.google.com/citations?user=XGJiEioAAAAJ&hl=en
https://aast.academia.edu/AhmedMagdy
Sentiment analysis and opinion mining is almost same thing however there is minor difference between them that is opinion mining extracts and analyze people's opinion about an entity while Sentiment analysis search for the sentiment words/expression in a text and then analyze it.
It uses machine learning techniques like SVM (Support Vector Machines) to analyze the text and classify them as positive, negative or neutral.
Sentiment analysis involves the process of automatically detecting the polarity of a text and extracting the author's reviews on the subject, and finally, classifying the text. In many research approaches, the textual data classification is done using deep learning models. This is due to the ability of deep learning models to classify a text with a high accuracy and the ability to model the sequence of textual data with word dependencies throughout the sentence. One of these deep learning models is RNN (Recurrent Neural Network). In order to use these models, the textual data and words must be converted into numerical vectors, for which various algorithms and methods have been proposed [10]. Today's pretrained word embedding libraries such as FastText have a high accuracy and quality in vector representations for words. Accordingly, in most current systems and research approaches, these libraries are used to convert the textual data to numerical vectors
II-SDV 2017: Localizing International Content for Search, Data Mining and Ana...Dr. Haxel Consult
Advances in text mining, analytics and machine learning are transforming our applications and enabling ever more powerful applications, yet most applications and platforms are designed to deal with a single (normalized) language. Hence as our applications and platforms are increasingly required to ingest international content, the challenge becomes to find ways to normalize content to a single language without compromising quality. An extension of this question in terms of such applications is also how we define quality in this context and what, if any, bi-products a localization effort can produce that may enhance the usefulness of the application.
This talk will, using patent searching as an example use case, review the challenges and possible solution approaches for handling localization effectively and will show what current emerging technology offers, what to expect and what not to expect and provide an introductory practical guide to handling localization in the context of data mining and analytics.
Enhancing Enterprise Search with Machine Learning - Simon Hughes, Dice.comSimon Hughes
In the talk I describe two approaches for improve the recall and precision of an enterprise search engine using machine learning techniques. The main focus is improving relevancy with ML while using your existing search stack, be that Luce, Solr, Elastic Search, Endeca or something else.
Slides presented at AI-Biz.
Title : Identifying Legality of Japanese Online Advertisements using Complex-valued Support Vector Machine with DFT-based Document Features
Mobile Recommendation Engine
collaborative filtering and content based approach in hybrid manner then Genetic Algorithm for Enhancement of the Recommendation Engine. by this marketers also will get the unique characteristics of the product that must be created and also recommend to the user.
Natural Language Processing: L01 introductionananth
This presentation introduces the course Natural Language Processing (NLP) by enumerating a number of applications, course positioning, challenges presented by Natural Language text and emerging approaches to topics like word representation.
Umm, how did you get that number? Managing Data Integrity throughout the Data...John Kinmonth
We live at the intersection of data and people. Data integrity is a function of the decisions that people make throughout the data lifecycle.
Dave De Noia, Pointmarc lead solution architect in data management, gives his take on the processes and people that affect data integrity throughout organizations at DRIVE 2014 (Data, Reporting, Intelligence, and Visualization Exchange)
Whether you're a retailer merging web analytics data with offline numbers or a healthcare company adding new data management software, De Noia explains how to avoid logic wobble and establish shared data structures.
About Dave:
Dave De Noia lives in the balance of chaos and order inherent to working with data. Starting his career at Microsoft building analyses in both SQL and big data environments, Dave later moved onto Redfin where he created and managed data infrastructure for analysis and reporting projects. Dave now serves as the senior solution and data architect at Pointmarc, a Bellevue-based digital analytics consultancy, where he helps some of the world’s largest brands get value from their data. Naturally functioning as a bridge between business and technical teams, Dave’s professional passion lies at the intersection of data and people.
About Pointmarc:
Pointmarc is a leading digital analytics agency providing actionable marketing insight and analytics platform instrumentation services for Fortune 500 clients within retail, technology, financial, media and pharmaceutical industries. With offices in Seattle, Boston, San Francisco and Portland, Pointmarc’s immersive approach to analytics empowers businesses to dive deeper into their data.
Email info@pointmarc.com for more information on data management or analytics instrumentation, and follow @pointmarc on Twitter for the latest in analytics.
Using AI to Build Fair and Equitable WorkplacesData Con LA
Data Con LA 2020
Description
With recent events putting a spotlight on anti-racism, social-justice, climate change, and mental health there's a call for increased ethics and transparency in business. Companies are, rightfully, feeling responsible for providing underrepresented employees with the same treatment and opportunities as their majority counterparts. AI can, and will, be used to help companies understand their environment, develop strategies for improvement and monitor progress. And, as AI is used to make increasingly complex and life-changing decisions, it is critical to ensure that these decisions are fair, equitable and explainable. Unfortunately, it is becoming increasingly clear that, much like humans, AI can be biased. It is therefore imperative that as we develop AI solutions, we are fully aware of the dangers of bias, understand how bias can manifest and know how to take steps to address and minimize it.
In this session you will learn:
*Definitions of fairness, regulated domains and protected classes
*How bias can manifest in AI
*How bias in AI can be measured, tracked and reduced
*Best practices for ensuring that bias doesn't creep into AI/ML models over time
*How explainability can be used to perform real-time checks on predictions
Speakers
Lawrence Spracklen, RSquared AI, Engineering Leadership
Sonya Balzer, RSquared.ai, Director of AI Marketing
SciBite is an award-winning leading provider of semantic solutions for the life sciences industry. Our fast, scalable easy-to-use semantic technologies understand the complexity and variability of content within life sciences. We can quickly identify and extract scientific terminology from unstructured text and transform it into valuable machine-readable data for your downstream applications. Our hand-curated ontologies ensure accuracy and reliability of high-quality results. Headquartered in the UK, we support our customers with additional sites in the US and Japan.
More infos at: www.scibite.com
Machine Learning Foundations for Professional ManagersAlbert Y. C. Chen
20180804@Taiwan AI Academy, Hsinchu
6 hour lecture for those new to machine learning, to grasps the concepts, advantages and limitations of various classical machine learning methods. More importantly, to learn the skills to break down large complicated AI projects into manageable pieces, where features and functionalities could be added incrementally and annotated data accumulated. Take home message: machine learning is always a delicate balance between model complexity M and number of data N so that the trained classifier generalizes well and does not overfit.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
2. Content
1. Background
2. Formulating the problem
3. Data Mining Process
4. Techniques
5. Analysis
01
3. What is Data
Mining?
• Extraction of meaningful / useful / Interesting
patterns from a large volume of data sources
• In this project, the source will be large
volume of WEB HOTEL REVIEWS data
• Data mining is one of the top ten emerging
technology
MIT’s TECHNOLOGY REVIEW 2004
4. What is Data
•
Mining?
Process of exploration and analysis
• By automatic / semi automatic means
• With little or no human interactions
• To discover meaningful patterns and rules
MASTERING DATA MINING BY BERRY AND LINOFF, 2000
5. User’s Opinions in Hotel
• Increase in social media and web
user
• Increase in valuable opinion
oriented data in Hotel due to web
expansion
• Identify potential hotel to stay by
looking at the aspects
• Overall Sentiments on hotel are
greatly sought on the web for
Sentiment Analysis
6. What can Data Mining do?
• Identify best prospects
(ASPECTS), and retain customers
• Predict what ASPECTS
customers like and promote
accordingly
• Learn parameters influencing
trends in sales and margins
• Identification of opinions for
customers
Sentiment Analysis !!!
7. What are the problems?
• Exponential growth of user’s
opinions
• Limitations of human analysis
• Accuracy of human analysis
Machines can be trained to take
over human analysis with advanced
computer technology and it is done
with LOW COST
8. Some Limitations of machines
• Unable to read like a human
• No emotions
• Cannot detect sarcasm
• Expression of sentiments in
different topic and domain
• Polarity analysis
• Facts Vs Opinion
9. Some machine limitation
• “The service is as good as none”.
examples
Negation not obvious to machine
• “Swimming pool is big enough to
swim with comfort” , “There is a
big crowd at the counter
complaining”. Polarity might
change with context.
• “The room is warmer than the
lobby”. Comparisons are hard to
classify
11. Machine Learning
• A tool for data mining and intelligent decision
support
• Application of computer algorithms that
improve automatically through experience
MASTERING DATA MINING BY BERRY AND LINOFF, 2000
12. Types of Machine learning
• Supervised Learning
• A training set is provided (data
with correct answers) which is
used to mine for known pattern
• Unsupervised Learning
• Data are provided with no prior
knowledge of the hidden
patterns that they contain.
• Semi Supervised Learning
14. Project Objective
• Prediction of sentence polarity
• Classification of polarity for sentiment
lexicon
• Detection of relations
15. Pre-requisite
• Large data set
• Relevant Prior Knowledge to
domain, in our case the hotel
domain
• Eg. Rating
• Sentiment lexicon for sentiment
analysis
• Data selection for reliability and
standards
17. Cleaning the “Dirty” Data (60% of
• effort)
Frequent problem : Data inconsistencies
• Duplicate data
• Spelling Errors != Trim from data
• Foreign accent and characters
• Singular / Plural conversion
• Punctuations removal / replacement
• Noise and incomplete data
• Naming convention misused, same name but
different meaning
19. Findings
• Part of Speech Tagging (POS) using Brill
Tagger - NO PROBLEM
-95% accuracy POS tagging words after data
cleaning
20. Findings
• Polarity tagging using sentiment lexicon –
BIG PROBLEM
-40% sentiment words not found in sentiment
lexicon
-10% sentiment words with a positive or
negative polarity found are in the neutral section
of sentiment lexicon
21. Problems
• Sentiment lexicon not comprehensive to fulfill
machine learning technique adopted
• Polarity of sentiment words who are domain
dependent are founded in neutral section of
sentiment lexicon
• Polarity of sentiment words can also change
within the domain even though they are
domain dependent
EXPANSION OF LEXICON !!!
22. Solution
• Classify the polarity of unlabeled sentiment
word using rule based mining
• Classify domain dependent sentiment words
• Establish word relations between labeled and
unlabeled sentiment words
23. Data Processing
• Rule based mining using conjunction and
punctuation
Polarity Assignment Rules
Same Adj – AND/OR - Adj
Opposite Neg - Adj – AND/OR - Adj /
Adj – AND/OR - Neg- Adj
Same Neg - Adj – AND/OR - Neg- Adj
Opposite Adj – BUT/NOR – Adj
Same Neg - Adj – BUT/NOR - Adj /
Adj – BUT/NOR - Neg- Adj
Opposite Neg - Adj – BUT/NOR - Neg- Adj
Same Adj , Adj
26. Analysis
• Using the expanded sentiment lexicon, we
analyze the polarity sentiment by doing a
sentiment lookup using Bayesian Network
27. Bayesian
• To determine polarity of sentiments
P(X | Y) = P(X) P(Y | X) / P(Y)
• Probability that a sentiments is positive or
negative, given it's contents
• Assumptions: There is no link between words
• P(sentiment | sentence) =
P(sentiment)P(sentence | sentiment) /
P(sentence)
28. Validation
• Precision = N (agree & found) / N (found)
• High precision means most of the correct
sentiment words are found by the system
• Recall = N (agree & found) / N (agree)
• High recall means most of found sentiment
words are correctly labeled by the system
29. Validation Results
• It is found that out of the 350 aspect-
unlabelled sentiment word pairs,
• Only 194 are founded by the methods.
Thus, the precision is about 57%.
• The recall is also not very high; only 126
words are corrected labelled by the
system, which is about 63%.
30. Discussion
• The results will improve if more rules are
applied such the inclusion of more adverbs
such as “excessively” as negation words.
• There might not be enough dataset for the
system to work on. There are only 350 aspect-
unlabelled sentiment word pairs for the
application to work with.
• This, however requires more human judges to
validate the data
31. Conclusion
• Comprehensive Sentiment Lexicon is a
simple yet effective solution to sentiment
analysis as it does not requires prior training
• Current sentiment lexicon does not capture
such domain and context sensitivities of
sentiment expressions
32. Conclusion
• This leads to poor coverage
• Thus, expanding general sentiment lexicon to
capture domain and context sensitivities of
sentiment expressions are advocated
What can data mining do in a hotel domain, in other words, learn the market
Impossible for humans to read every single opinionsBiased of humans to read certain opinionsMachinesAllow fast access to vast amount of dataAllow computational intensive algorithm and statistical methods
Impossible for humans to read every single opinionsBiased of humans to read certain opinionsMachinesAllow fast access to vast amount of dataAllow computational intensive algorithm and statistical methods
Many fields of data mining and in this project we will focus on these 4
Growing data volume , limitation of humans and low cost to human
The goal for unsupervised learning is to discover these patternsSemi – Knowledge is known and applied from one data collection in order to mine, classify, analyze, interpret a related data collection
Some of the problems to be solved by data miningPrediction of sentence polarityClassification of polarity for sentiment lexiconDetection of relations
Data inconsistencies: Say good in the title but in the review say bad
Assigning a label to every word in the text to allow machine to do something with it
Pos tagging wrong due to some word like heart having double tagging
For example, in the domain of handheld devices, the word “large” can express positivity for screen size but negativity in the phone size.
Assigning a label to every word in the text to allow machine to do something with it
After establishing relations, we have a graph of nodes (Sentiments / Aspects)Determine the probability that the node is positive or negative given its surrounding nodesStart with a high frequency unlabelled sentiment word-aspect pair then based on the aspect and its label semtiment pair, determine the polarity for the unlabelThis process iterate till all unlabe found their polarity
After establishing relations, we have a graph of nodes (Sentiments / Aspects)Determine the probability that the node is positive or negative given its surrounding nodesStart with a high frequency unlabelled sentiment word-aspect pair then based on the aspect and its label semtiment pair, determine the polarity for the unlabelThis process iterate till all unlabe found their polarity
Assigning a label to every word in the text to allow machine to do something with it
A comprehensive sentiment lexicon can provide a simple yet effective solution to sentiment analysis, because it is general and does not require prior training. Therefore, attention and effort have been paid to the construction of such lexicons. However, a significant challenge to this approach is that the polarity of many words is domain and context dependent. For example, ‘long’ is positive in ‘long battery life’ and negative in ‘long shutter lag.’ Current sentiment lexicons do not capture such domain and context sensitivities of sentiment expressions. They either exclude such domain and context dependent sentiment expressions or tag them with an overall polarity tendency based on statistics gathered from certain corpus such as the world wide web accessed via the internet. While excluding such expressions leads to poor coverage, simply tagging them with a polarity tendency leads to poor precision.
ATheyeither exclude such domain and context dependent sentiment expressions or tag them with an overall polarity tendency based on statistics gathered from certain corpus such as the world wide web accessed via the internet. While excluding such expressions leads to poor coverage, simply tagging them with a polarity tendency leads to poor precision.