Sais svcc
•Download as PPTX, PDF•
1 like•1,917 views
This document discusses sentiment analysis and how it can be applied using Apache Solr and UIMA. It provides an overview of sentiment analysis and defines key concepts. It then describes how Solr can be used to build search indexes and how UIMA can be used to integrate natural language processing tools for tasks like entity extraction and sentiment analysis. The document concludes with proposing a sentiment analysis application that uses various NLP techniques within Solr and UIMA to classify sentiment in text.
Recommended
More Related Content
Similar to Sais svcc
Similar to Sais svcc (20)
Recently uploaded
Recently uploaded (20)
Sais svcc
- 2. Sentiment Analysis? Sentiment Analysis – General Architecture Little Lucene Sentiment Analysis and Solr Applications of Sentiment Analysis Code Walkthrough
- 3. Working mostly in Search domain Search = IR + ML + NLP Who am I? Works for
- 4. Contributing to SolrSherlock - Open Source Project Who am I? http://solrsherlock.github.io/SolrSherlock/
- 5. What is Sentiment Analysis? A linguistic analysis technique that identifies The movie is great. The movie stars Mr. X The movie is horrible. opinion early in a piece of text.
- 6. Challenging Too easy Too hard Difficulty misclassification What is Sentiment Analysis?
- 7. Sentiment Analysis NLP Cognitive Science What is Sentiment Analysis?
- 8. Human can easily understand emotions. Can a machine be trained to do it? What is Sentiment Analysis?
- 9. SA offers organizations ability to monitor in real time and act accordingly Marketing managers, PR Firms, campaign managers, politicians, equity investors, on line shoppers are direct beneficiaries http://www.tweetfeel.com http://www.nytimes.com/interactive/us/pol itics/2010-twitter-candidates.html
- 11. Document-Level supervised/non supervised learning Sentence-Level supervised learning Feature-Based Sentiment Analysis All NP in corpus and Polarity Sentiment Lexicon Acquisition WordNet
- 12. Open-source Java based search engine Provides document indexing w/ arbitrary fields and fast search Several relevance and ranking algorithms
- 13. 1. Create an index 2. Add ‘document’ representations of items 3. Construct queries 4. Ask for results (will be scored )
- 14. IndexWriterConfig config = /* configure */ ; Directory dir = FSDirectory.open(indexFile); IndexWriter w = new IndexWriter(dir, config); for (ItemInfo item: getItems()) { Document doc = new Document(); doc.add(new Field("title", item.title)); doc.add(new Field("tags", item.tags)); w.add(doc); } w.close();
- 15. IndexSearcher idx = getIndexSearcher(); IndexReader reader = idx.getIndexReader(); TopDocs results = idx.search(q, n + 1);
- 16. PyLucene is Python implementation Lucy is in C w/ bindings for other langs Lucene.NET SOLR provides search server (with REST API) on top of Lucene
- 17. Solr ? Http Request Servlet Admin Interface Update Servlet Standard Request Handler Custom Request Handler Response Writer Solr Core Lucene Analysis UIMA config Caching Update Handler
- 18. Linguistics module Stems, Lemmas and Synonyms multi language capability CJKAnalyzer, UIMA Analyzers UIMA integration UpdateProcessorChain Why Solr ?
- 19. Why Solr ? Extract domain specific entities and concepts Time and Cost Solr Set Up – 5 mins UIMA Annotators - 5 days Enrich text, write to dedicated field
- 20. Tagging entities in review text Applications: I wasn't really in the market for another tablet, but my girlfriend ended up getting one for me so she got me on this one. I would like to say that this tablet reminds me of the first Motorola Droid smartphone that came out several years back. The phone jam packed a ton of bells & whistles into its hardware and software to give a lot of bang for your buck. This is what it feels like amazon has done with the Kindle Fire 8.9. They have put a lot of advanced hardware and innovative software, so for the average user, specially someone who absorbs a lot of media, you get a lot for the price. But just because you get a lot for the price, doesn't mean it is without its flaws.
- 21. Applications: Consumer feedback about products Which product features are more relevant Polarity
- 22. Digital SLR with Full 1080p HD Video There are many preprogrammed scene modes that make this a very easy camera to use. The picture quality is beyond belief, and even better for the price. Price: Usecase
- 23. Why UIMA ? UIMA Framework manages components and data flow – No coding Deploy pipeline of analysis engines AEs wrap NLP algorithms Person Place organization Language Detection Aggregate analysis engine Sentence Annotator POS Annotator NER
- 25. NLP+UIMA Use POS in query understanding boosting terms Synonym expansion Extract concepts/entities Faceting using entities Identify places in query and use spatial queries
- 26. Ideas: Sentiment Analysis App Identify Subjective Sentences from text Remove noisy sentences – Regex, conditional probability Graph min cut – LingPipe Subjectivity Lexicons Discard Facts and Objective Sentences
- 28. Sentiments Intensity - SentiWordNet WordNet-Affect: WordNet + annotated concepts Ideas: Sentiment Analysis App Hybrid model with adding dictionary
- 29. Update Handler with processor chain Remove Duplicates processor Logging processor Custom Transform processor Index processor Update Processor Chain Text Analyzers Lucene Lucene Index Sentence Detection processor Sentiment Classifier Company Name Annotator Sentiment Score processor Product Reviews
- 30. Let’s look at the code
- 31. Data transformation or post processing UpdateProcessorFactory LogUpdateProcessorFactory UIMAUpdateProcessorFactory UpdateRequestProcessorChain ◦ Pipe line of UpdateRequestProcessors
- 32. <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" > <lst name="defaults"> <str name="update.processor">uima</str> </lst> </requestHandler>
- 33. Stanford NER
- 34. <updateRequestProcessorChain name="uima"> <processor class="org.apache.solr.uima.processor.UIMAUpdateRequestProcessorFactory"> <lst name="uimaConfig"> <lst name="runtimeParameters"> </lst> <lst name="analysisEngine"><str name="defaultanalysisEngine">/org/apache/uima/desc/OverridingParamsExtServicesAE.xml</str> </lst> <lst name="analyzeFields"> <bool name="merge">false</bool> <arr name="fields"> <str>content_text</str> </arr> </lst> <lst name="fieldMappings"> <lst name="type"> <str name="name">org.apache.uima.DictionaryEntry</str> <lst name="mapping"> <str name="feature">coveredText</str> <str name="field">sentiment_keyword,sentiment_type</str> </lst> </lst>
- 37. Questions ?
Editor's Notes
- Huge explosion today of sentiments. PR firms, etc r Direct beneficiary of SA technology
- Classify sentences into 2 principal classes subjective, objective
- Positive, negative neutral, naïve bayes
- Overall this review is very positive about smart phone, sentiment score
- Using hierarchical classification, neutrality is determined first, and sentiment polarity is determined second, but only if the text is not neutral.
- No matter how you choose to import data, there is a final config point within solr that allows manipulation of the imported data before it get indexed. Updaterquesthandler put documents on an update request processor chain.