CountVectorizer in Flink

1. CountVectorizer in Flink
Insight Data Engineering Fellowship, Silicon Valley
Roshani Nagmote

2. Motivation
• Open Source Contribu?on to Apache Flink
• Add CountVectorizer in Flink ML library to help NLP
• Usage : Documents similarity / Word Cloud applica?on

3. What is CountVectorizer
• Class in Python Scikit-learn, ML library

4. Functionality Implemented
• fit() (Es?mator) :
[“this is a text”,
“this is not a text
document to
document”]
Output of fit
(this,1) (is,2) (a,3)
(text,4) (not,5)
(document,6) (to,
7)

5. Functionality Implemented
• transform() (Transformer) :
• get_feature_names() :
Output
(this,1) (is,2) (a,3)
(text,4) (not,5)
(document,6) (to,
7)
[“this is a text”,
“this is not a text
document to
document”]
Output of Transform
[(1,1.0) (2,1.0) (3,1.0)
(4,1.0)]
[(1,1.0) (2,1.0) (5,1.0)
(3,1.0) (4,1.0) (6,2.0)
(7,1.0)]

6. Functionality Implemented
• Parameters added to countVectorizer() constructor
-> setMinDF(2), setMaxDF(5)
Mindf =2 The sun in the sky is bright. We can see the shining sun, the bright sun
Maxdf = 5 The sun in the sky is bright. We can see the shining sun, the bright sun
-> setStopwords (List([“in”,”the”] ))
The sun in the sky is bright. We can see the shining sun, the bright sun

7. Functionality Implemented
• Parameters
-> setNgramRange(List(1,3))
• Wri?ng test cases to test above func?ons and compare output
with scikit-learn countvectorizer for accuracy - 100%

8. Sample Output

9. Sample Output

10. Overview
CountVectorizer()
setStopwords setMinDF
setMaxDF setNgramRange
3
10
List[“is”,”the”]
List[1,3]

11. Overview
CountVectorizer() Fit()
Input
Data
files
setStopwords setMinDF
setMaxDF setNgramRange
[“this is a text”,
“this is not a text
document to
document”] 3
10
List[“is”,”the”]
List[1,3]
Output of fit
(this,1) (is,2) (a,3)
(text,4) (not,5)
(document,6) (to,7)

12. Overview
CountVectorizer() Fit()
Transform
Input
Data
files
setStopwords
setMinDF
setMaxDF setNgramRange
Output of fit
(this,1) (is,2) (a,3)
(text,4) (not,5)
(document,6) (to,7)
[“this is a text”,
“this is not a text
document to
document”] Output of Transform
[(1,1.0) (2,1.0) (3,1.0)
(4,1.0)]
[(1,1.0) (2,1.0) (5,1.0)
(3,1.0) (4,1.0) (6,2.0)
(7,1.0)]
3
10
List[“is”,”the”]
List[1,3]

13. Overview
CountVectorizer() Fit()
Transform
Input
Data
files
setStopwords
setMinDF
setMaxDF setNgramRange
getFeature
Names
Output of fit
(this,1) (is,2) (a,3)
(text,4) (not,5)
(document,6) (to,7)
[“this is a text”,
“this is not a text
document to
document”] Output of Transform
[(1,1.0) (2,1.0) (3,1.0)
(4,1.0)]
[(1,1.0) (2,1.0) (5,1.0)
(3,1.0) (4,1.0) (6,2.0)
(7,1.0)]
3
10
List[“is”,”the”]
List[1,3]

14. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
GetParameters
Call
Fit(text)

15. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
GetParameters
Call
Fit(text)
Validate
Parameters

16. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
Train
Dataset
Func?on
GetParameters
Call
Fit(text)
Validate
Parameters

17. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
Train
Dataset
Func?on
GetParameters
Call
Fit(text)
Validate
Parameters
RegexParsing

18. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
Train
Dataset
Func?on
GetParameters
Call
Fit(text)
Validate
Parameters
Filter data on
mindf,maxdf
RegexParsing

19. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
Train
Dataset
Func?on
GetParameters
Call
Fit(text)
Validate
Parameters
Filter data on
mindf,maxdf
RegexParsing
Filter data on
stopwords

20. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
Train
Dataset
Func?on
GetParameters
Call
Fit(text)
Validate
Parameters
Filter data on
mindf,maxdf
RegexParsing
Filter data on
stopwords
Create tuples
of ngram range

21. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
Train
Dataset
Func?on
GetParameters
Call
Fit(text)
Validate
Parameters
Filter data on
mindf,maxdf
RegexParsing
Filter data on
stopwords
Create tuples
of ngram range
Add in hashset

22. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
Train
Dataset
Func?on
GetParameters
Call
Fit(text)
Validate
Parameters
Filter data on
mindf,maxdf
RegexParsing
Filter data on
stopwords
Create tuples
of ngram range
Add in hashset
Zipwithindex
Output

23. Fit(Text) function
Fit()
Override fit
from
Es?mator
All
docum
ents
Train
Dataset
Func?on
GetParameters
Call
Fit(text)
Validate
Parameters
Filter data on
mindf,maxdf
RegexParsing
Filter data on
stopwords
Create tuples
of ngram range
Add in hashset
Zipwithindex
Map
ReduceGroup
Output
Flink
Operators

24. Transform(Text)
Transform()
Override
from
Transformer
Model
(Fit Output)
Document
Provide Req
Parameters

25. Transform(Text)
Transform()
Override
from
Transformer
Model
(Fit Output)
Document
Transform
Dataset
Func?on
Provide Req
Parameters

26. Transform(Text)
Transform()
Override
from
Transformer
Model
(Fit Output)
Document
Transform
Dataset
Func?on
RegexParsing
Provide Req
Parameters

27. Transform(Text)
Transform()
Override
from
Transformer
Model
(Fit Output)
Document
Transform
Dataset
Func?on
RegexParsing
Create tuples of
ngram range
Provide Req
Parameters

28. Transform(Text)
Transform()
Override
from
Transformer
Model
(Fit Output)
Document
Transform
Dataset
Func?on
RegexParsing
Create tuples of
ngram range
Matching
(index,1.0)
Provide Req
Parameters

29. Transform(Text)
Transform()
Override
from
Transformer
Model
(Fit Output)
Document
Transform
Dataset
Func?on
RegexParsing
Create tuples of
ngram range
Matching
(index,1.0)
Convert to
sparse vector
Sparse vector
(Flink func?on)
Output
Internally uses
quicksort and duplicate
entries are summed to
get frequency
Provide Req
Parameters

30. Word Cloud
• Used Lightning Scala api for visualiza?on purpose
• hgp://countvectorizer.us

31. About Me
Roshani Nagmote
● Masters in Computer Science from
University of Utah
● Data Plaiorm Engineering intern at Ask.com
● Web Developer at Tata Consultancy Services
Things I love:
● Playing with my Rini
● Hiking and visi?ng new places