Pune Developer's Community Meetup on Text Mining - Text data Visualization by Manjusha Joshi

1. #AI2018
Text Mining - Text data Visualization
Manjusha Joshi
Sr. Data Scientist
Maxima IT Consulting
June 23, 2018
manjusha.joshi@gmail.com

2. #AI2018Overview
Data
Sources of text Data
Information Retrival
Text data : Techniques
Text Data Mining
Challeges in Text Data Mining
Text Data: AI
Text data visualization

3. #AI2018Types of data
Numerical

4. #AI2018Types of data
Numerical
Text

5. #AI2018Sources of text Data
Social media: Facebook, tweeter

6. #AI2018Sources of text Data
Social media: Facebook, tweeter
emails, blogs

7. #AI2018Sources of text Data
Social media: Facebook, tweeter
emails, blogs
Structured text : Ex. Book pdf, research papers

8. #AI2018OCR
Scanned document OCR

9. #AI2018Information Retrival
Numerical data: Max, Min, Average

10. #AI2018Information Retrival
Numerical data: Max, Min, Average
Text Data: ?

11. #AI2018Information Retrival
Numerical data: Max, Min, Average
Text Data: ?

12. #AI2018Information Retrival
Numerical data: Max, Min, Average
Text Data: ?

13. #AI2018

14. #AI2018Text data : Techniques
Term Freduency: TF

15. #AI2018Text data : Techniques
Term Freduency: TF
Document Frequency: DF

16. #AI2018Text data : Techniques
Term Freduency: TF
Document Frequency: DF
Inverse DF: IDF

17. #AI2018Text data : Techniques
Term Freduency: TF
Document Frequency: DF
Inverse DF: IDF
TF-IDF

18. #AI2018Text data : Techniques
Term Freduency: TF
Document Frequency: DF
Inverse DF: IDF
TF-IDF

19. #AI2018Text data : Techniques
Term Freduency: TF
Document Frequency: DF
Inverse DF: IDF
TF-IDF

20. #AI2018TF-IDF

21. #AI2018Text data : Techniques
NLP

22. #AI2018Text data : Techniques
NLP
Tokenization, n-gram

23. #AI2018Text data : Techniques
NLP
Tokenization, n-gram

24. #AI2018Text data : Techniques
NLP
Tokenization, n-gram Tokenization: Given a character
sequence and a defined document unit, tokenization is the
task of chopping it up into pieces, called tokens

25. #AI2018Text data : Techniques
NLP
Tokenization, n-gram Tokenization: Given a character
sequence and a defined document unit, tokenization is the
task of chopping it up into pieces, called tokens
Bag of words

26. #AI2018Text data : Techniques
NLP
Tokenization, n-gram Tokenization: Given a character
sequence and a defined document unit, tokenization is the
task of chopping it up into pieces, called tokens
Bag of words
word2Vec

27. #AI2018Text data : Techniques
NLP
Tokenization, n-gram Tokenization: Given a character
sequence and a defined document unit, tokenization is the
task of chopping it up into pieces, called tokens
Bag of words
word2Vec
Name Entity Extraction

28. #AI2018Text data : Techniques
NLP
Tokenization, n-gram Tokenization: Given a character
sequence and a defined document unit, tokenization is the
task of chopping it up into pieces, called tokens
Bag of words
word2Vec
Name Entity Extraction

29. #AI2018Text data : Techniques
NLP
Tokenization, n-gram Tokenization: Given a character
sequence and a defined document unit, tokenization is the
task of chopping it up into pieces, called tokens
Bag of words
word2Vec
Name Entity Extraction
Classify named entities in text into pre-defined categories such
as the names of persons, organizations, locations etc.

30. #AI2018Text Data Mining
Find require information: Query

31. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords

32. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions

33. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions

34. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden

35. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden Angry mad, furious, enraged, excited, wrathful, aroused,
inflamed

36. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden Angry mad, furious, enraged, excited, wrathful, aroused,
inflamed Calm quiet, peaceful, still, tranquil, mild, serene,
smooth, unexcited, detached, aloof

37. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden Angry mad, furious, enraged, excited, wrathful, aroused,
inflamed Calm quiet, peaceful, still, tranquil, mild, serene,
smooth, unexcited, detached, aloof Eager keen, fervent,
enthusiastic, involved, interested, alive to

38. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden Angry mad, furious, enraged, excited, wrathful, aroused,
inflamed Calm quiet, peaceful, still, tranquil, mild, serene,
smooth, unexcited, detached, aloof Eager keen, fervent,
enthusiastic, involved, interested, alive to Fear fright, dread,
terror, alarm, dismay, anxiety, scare, horror, panic, apprehension

39. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden Angry mad, furious, enraged, excited, wrathful, aroused,
inflamed Calm quiet, peaceful, still, tranquil, mild, serene,
smooth, unexcited, detached, aloof Eager keen, fervent,
enthusiastic, involved, interested, alive to Fear fright, dread,
terror, alarm, dismay, anxiety, scare, horror, panic, apprehension
Happy pleased, contented, satisfied, delighted, elated, joyful,
cheerful, glad, blissful, overjoyed

40. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden Angry mad, furious, enraged, excited, wrathful, aroused,
inflamed Calm quiet, peaceful, still, tranquil, mild, serene,
smooth, unexcited, detached, aloof Eager keen, fervent,
enthusiastic, involved, interested, alive to Fear fright, dread,
terror, alarm, dismay, anxiety, scare, horror, panic, apprehension
Happy pleased, contented, satisfied, delighted, elated, joyful,
cheerful, glad, blissful, overjoyed Hate despise, loathe, detest,
abhor, disfavor, dislike, disapprove

41. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden Angry mad, furious, enraged, excited, wrathful, aroused,
inflamed Calm quiet, peaceful, still, tranquil, mild, serene,
smooth, unexcited, detached, aloof Eager keen, fervent,
enthusiastic, involved, interested, alive to Fear fright, dread,
terror, alarm, dismay, anxiety, scare, horror, panic, apprehension
Happy pleased, contented, satisfied, delighted, elated, joyful,
cheerful, glad, blissful, overjoyed Hate despise, loathe, detest,
abhor, disfavor, dislike, disapprove Love like, admire, esteem,
fancy, care for, cherish, adore, treasure, worship, appreciate, savor

42. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden Angry mad, furious, enraged, excited, wrathful, aroused,
inflamed Calm quiet, peaceful, still, tranquil, mild, serene,
smooth, unexcited, detached, aloof Eager keen, fervent,
enthusiastic, involved, interested, alive to Fear fright, dread,
terror, alarm, dismay, anxiety, scare, horror, panic, apprehension
Happy pleased, contented, satisfied, delighted, elated, joyful,
cheerful, glad, blissful, overjoyed Hate despise, loathe, detest,
abhor, disfavor, dislike, disapprove Love like, admire, esteem,
fancy, care for, cherish, adore, treasure, worship, appreciate, savor
Sad miserable, poor, sorrowful, depressed, glum, gloomy, dismal,
discouraged, unhappy

43. #AI2018Text Data Mining
Find require information: Query
Find important words: Keywords
Customer satisfaction: emotions
Feelings
Anger enrage, infuriate, arouse, nettle, exasperate, inflame,
madden Angry mad, furious, enraged, excited, wrathful, aroused,
inflamed Calm quiet, peaceful, still, tranquil, mild, serene,
smooth, unexcited, detached, aloof Eager keen, fervent,
enthusiastic, involved, interested, alive to Fear fright, dread,
terror, alarm, dismay, anxiety, scare, horror, panic, apprehension
Happy pleased, contented, satisfied, delighted, elated, joyful,
cheerful, glad, blissful, overjoyed Hate despise, loathe, detest,
abhor, disfavor, dislike, disapprove Love like, admire, esteem,
fancy, care for, cherish, adore, treasure, worship, appreciate, savor
Sad miserable, poor, sorrowful, depressed, glum, gloomy, dismal,
discouraged, unhappy Scared afraid, frightened, alarmed, terrified,
panicked, fearful, shy, skittish, jumpy, disquieted, worried

44. #AI2018Challeges in Text Data Mining
Missing data: s me − > some / same ?

45. #AI2018Challeges in Text Data Mining
Missing data: s me − > some / same ?
Data with garbadge : scanned data with non English
characters @ etc.

46. #AI2018Challeges in Text Data Mining
Missing data: s me − > some / same ?
Data with garbadge : scanned data with non English
characters @ etc.
Incorrect data: Typo mistakes

47. #AI2018Challeges in Text Data Mining
Missing data: s me − > some / same ?
Data with garbadge : scanned data with non English
characters @ etc.
Incorrect data: Typo mistakes
Words with same meaning: Synonym Ex. Happy , pleased,
delighted, glad

48. #AI2018Challeges in Text Data Mining
Missing data: s me − > some / same ?
Data with garbadge : scanned data with non English
characters @ etc.
Incorrect data: Typo mistakes
Words with same meaning: Synonym Ex. Happy , pleased,
delighted, glad
Words with mulitple meaning: Homonyms Right: You were
right. Make a right turn at the tree.

49. #AI2018Text Data: AI
Predict next word/phrase

50. #AI2018Text Data: AI
Predict next word/phrase
BoT :

51. #AI2018Text Data: AI
Predict next word/phrase
BoT :

52. #AI2018Text Data: AI
Predict next word/phrase
BoT : Robot

53. #AI2018Text Data: AI
Predict next word/phrase
BoT : Robot An Internet Bot, also known as web robot,
WWW robot or simply -bot-, is a software application that
runs automated tasks (scripts) over the Internet.
Topic Modelling

54. #AI2018Text Data: AI
Predict next word/phrase
BoT : Robot An Internet Bot, also known as web robot,
WWW robot or simply -bot-, is a software application that
runs automated tasks (scripts) over the Internet.
Topic Modelling

55. #AI2018Text Data: AI
Predict next word/phrase
BoT : Robot An Internet Bot, also known as web robot,
WWW robot or simply -bot-, is a software application that
runs automated tasks (scripts) over the Internet.
Topic Modelling

56. #AI2018Text Data: AI
Predict next word/phrase
BoT : Robot An Internet Bot, also known as web robot,
WWW robot or simply -bot-, is a software application that
runs automated tasks (scripts) over the Internet.
Topic Modelling
Product recommendation

57. #AI2018Text Data: AI
Predict next word/phrase
BoT : Robot An Internet Bot, also known as web robot,
WWW robot or simply -bot-, is a software application that
runs automated tasks (scripts) over the Internet.
Topic Modelling
Product recommendation
Smart Document summerization

58. #AI2018Text Data: AI
Predict next word/phrase
BoT : Robot An Internet Bot, also known as web robot,
WWW robot or simply -bot-, is a software application that
runs automated tasks (scripts) over the Internet.
Topic Modelling
Product recommendation
Smart Document summerization
Smart categorisation of documents Ex. Gmail spam mails

59. #AI2018Text data visualization :WordCloud

60. #AI2018WordCloud

61. #AI2018Data Tree

62. #AI2018Data Tree

63. #AI2018Text Network

64. #AI2018Text Connectivity