NLP processing tricks

1. Natural Language Processing

2. Fundamental
Tasks
 1. Splitting Sentences/Word Tokenizing
 2.Parts of speech tagging
 3.Named Entity recognition
 4.Stemming/Lemmatization
 5.Semantic Text Similarity
 6.Topic Modelling
 7.Text Clustering
 8.Sentiment Analysis/Text Classification
 9.Word embedding

3. Applications
 1.Machine Translation
 2.Image/Video Captioning
 3.Image Question Answering
 4.Textual Entailment(TE)
 5.Transliteration
 6.Document Summarization
 7.Dialog
 8.NLU

4. Named Entity
Recognition
https://www.paralleldots.com/named-entity-recognition
http://nlp.stanford.edu:8080/ner/

5. Sentiment
Analysis

6. Text
classification

7. Text
suggestions

8. TextClustering

9. Topic
Modelling

10. Machine
Translation

11. Transliteration

12. Image
Captioning

13. Textual
Entailment

14. Document
Summarization

15. Dialog/AI
Chatbot

16. NLU –Natural
language
Understanding
 IBMWATSON DISCOVERY
 FACEBOOK BABI
https://research.fb.com/downloads/babi/

17. Python
Projects
1. NLTK
2.Spacy
3.Gensim
4.Pyspotlight
5. FRED
6.pyLDAvis

18. POSTagging
Using NLTK
 The Natural LanguageToolkit

19. Named Entity

20. Named Entity
recognition
 NLTK

21. Entity Linking
https://dbpedia.org/sparql
https://www.w3.org/2009/Talks/0615-qbe/
https://rdflib.github.io/sparqlwrapper/
http://wit.istc.cnr.it/stlab-tools/fred/demo/

22. Stemming

23. Lemmatization

24. Sentence
similarity
 Documents are converted to vectors using tf-idf vectorizer
 Similarity model is created using this tf-idf vector and this is used
to calculate similarity score for a new document.

25. Topic
Modelling
LDA – Latest Dirichelet Allocation
-Unsupervised Learning, we have to choose number of topics
ahead of time
-Each document is represented as a distribution over topic
-Each topic is represented as a distribution over words.
Algorithm convert each document into a vector of dimension equals
to number of topic and the each value of vector would give how
much the document goes with that topic.

26. Topic
Modelling
Code

27. Visualizing
LDA

28. Word2Vec
 Dense representation, each word vectors are usually 300 long
 Created using skip-gram algorithm

29. Word2Vec