This document discusses using Apache Spark and MLlib for text mining on big data. It outlines common text mining applications, describes how Spark and MLlib enable scalable machine learning on large datasets, and provides examples of text mining workflows and pipelines that can be built with Spark MLlib algorithms and components like tokenization, feature extraction, and modeling. It also discusses customizing ML pipelines and the Zeppelin notebook platform for collaborative data science work.

1. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
RevolutionizeTextMining
withSparkandZeppelin
April2017
YanboLiang
ApacheSparkcommitter
Softwareengineer@Hortonworks

2. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Agenda
TextminingworkflowonBigData
TextminingwithSparkandMLlib
SparkandZeppelinastheplatform

3. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
TextMining:PracticalApplications
•Textclassification
–Spamfiltering
–Frauddetection
•Textclustering
•Sentimentanalysis
•Entityextraction
•Recommendations
•Automaticlabeling
•Contextualadvertising

4. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
TraditionalTextMining
•Commercialsoftware
•Opensourcesoftware
–Gensim,KNIME,NLTK,
sklearn,R

5. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
TraditionalTextMining
•Commercialsoftware
–IBMSPSS,RapidMiner,SAS
•Opensourcesoftware
–Gensim,KNIME,NLTK,
sklearn,R

6. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
TextMiningonBigData

7. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
TextMiningonBigData
DataScientistsSoftwareengineers

8. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
WhyApacheSparkMLlib
•ScalablemachinelearningalgorithmsontopofSpark
–AlternatingLeastSquaresonSpotifydata
•50+millionusersx30+millionsongs,50billionratings
•Forrank10with10iterations,~1hourrunningtime
•Workflowutilities
–MLpipeline
–Modelimport/export
–crossvalidation

9. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
TextMiningworkflow
•Prototype(Python/R)
•CreatePipeline
–Loaddataset
–Extractrawfeatures
–Transformfeatures
–Selectkeyfeatures
–Fitandchoosebestmodels
•Re-implementPipelinefor
production(Java/Scala)
•DeployPipeline
•Scoring
DataScienceSoftwareengineering

10. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
TextMiningworkflow
•Prototype(Python/R)
•CreatePipeline
–Loaddataset
–Extractrawfeatures
–Transformfeatures
–Selectkeyfeatures
–Fitandchoosebestmodels
•Re-implementPipelinefor
production(Java/Scala)
•DeployPipeline
•Scoring
DataScienceSoftwareengineering

11. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Loaddata
TextLabel
Iboughtthegame…4
DoNOTbothertry…1
Thisshirtisawesome…5
nevergotit.Seller…1
Iorderedthisto…3
Dataset
Feature
engineering
Model
training
Model
evaluation

12. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Extractfeatures
TextLabelWordsFeatures
Iboughtthegame…4“i”,“bought”,…[1,0,3,9,…]
DoNOTbothertry…1“do”,“not”,…[0,0,11,0,…]
Thisshirtisawesome…5“this”,“shirt”,…[0,2,3,1,…]
nevergotit.Seller…1“never”,“got”,…[1,2,0,0,…]
Iorderedthisto…3“i”,“ordered”,…[1,0,0,3,…]
Dataset
Feature
engineering
Model
training
Model
evaluation

13. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Fitamodel
TextLabelWordsFeaturesProbabilityPrediction
Iboughtthegame…4“i”,“bought”,…[1,0,3,9,…]0.84
DoNOTbothertry…1“do”,“not”,…[0,0,11,0,…]0.62
Thisshirtisawesome…5“this”,“shirt”,…[0,2,3,1,…]0.95
nevergotit.Seller…1“never”,“got”,…[1,2,0,0,…]0.71
Iorderedthisto…3“i”,“ordered”,…[1,0,0,3,…]0.74
Dataset
Feature
engineering
Model
training
Model
evaluation

14. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Evaluate
TextLabelWordsFeaturesProbabilityPrediction
Iboughtthegame…4“i”,“bought”,…[1,0,3,9,…]0.84
DoNOTbothertry…1“do”,“not”,…[0,0,11,0,…]0.62
Thisshirtisawesome…5“this”,“shirt”,…[0,2,3,1,…]0.95
nevergotit.Seller…1“never”,“got”,…[1,2,0,0,…]0.71
Iorderedthisto…3“i”,“ordered”,…[1,0,0,3,…]0.74
Dataset
Feature
engineering
Model
training
Model
evaluation

15. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
KeyabstractionofSparkMLpipeline
•Transformer
–Featuretransformers(e.g.,HashingTF)andtrainedMLmodels(e.g.,NaiveBayesModel).
•Estimator
–MLalgorithmsfortrainingmodels(e.g.,NaiveBayes).
•Evaluator
–Theseevaluatepredictionsandcomputemetrics,usefulfortuningalgorithmparameters(e.g.,
BinaryClassificationEvaluator).

16. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Spark’sTextMiningalgorithms
•LDAfortopicmodel
•Word2Vecanunsupervisedwaytoturnwordsintofeaturesbasedontheirmeaning
•CountVectorizerturnsdocumentsintovectorsbasedonwordcount
•HashingTF-IDFcalculatesimportantwordsofadocumentwithrespecttothecorpus
•Andmuchmore

17. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
MLlibTextMiningPipeline-classification
Dataset
RegexTokenizer
StopWordsRemover
CountVectorizer
HashingTF
IDF
StringIndexer
NaiveBayes
LogisticRegression
SVM
MLP
textclassification

18. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
MLlibTextMiningPipeline–topicmodel
Dataset
RegexTokenizer
StopWordsRemover
CountVectorizer
HashingTF
IDFLDAtopicmodel

19. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
MLlibTextMiningPipeline-recommendation
Dataset
RegexTokenizerWord2Vec
recommendation

20. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
MLlibTextMiningPipeline
Dataset
RegexTokenizer
StopWordsRemover
CountVectorizer
HashingTF
IDF
StringIndexer
NaiveBayes
LogisticRegression
SVM
MLP
LDA
Word2Vec
textclassification
topicmodel
recommendation

21. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Demo
•loadthefilecontentsandthecategories
•extractfeaturevectorssuitableformachinelearning
•trainalinearmodeltoperformcategorization
•useagridsearchstrategytofindagoodconfigurationofboththefeatureextraction
componentsandtheclassifier
https://github.com/yanboliang/dataworks-munich-2017

22. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
CustomingMLPipelines
•MLlib2.1includes:
–30+featuretransformers(Tokenizer,Word2Vec,…)
–25+models(forclassification,regression,clustering,…)
–Modeltuning&evaluation
•Butsomeapplicationsrequirecustomized
–Transformers&Models

23. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Optionsforcustomization
•Existingusecases:
–spark-corenlp
–spark-vlbfgs
•Extendabstractions
–Transformer
–Estimator&Model
–Evaluator

24. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Sparkvirtualenvironment
DataScientistADataScientistB
Python2.7
Python2.7
Python2.7
Python2.7
Python2.7

25. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Sparkvirtualenvironment
DataScientistADataScientistB
Python2.7
Python2.7
Python2.7
Python2.7
Python2.7
Python3.5
Python3.5
Python3.5

26. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
TextMiningworkflow
•Prototype(Python/R)
•CreatePipeline
–Loaddataset
–Extractrawfeatures
–Transformfeatures
–Selectkeyfeatures
–Fitandchoosebestmodels
•Re-implementPipelinefor
production(Java/Scala)
•DeployPipeline
•Scoring
DataScienceSoftwareengineering
Duplicatedand
error-prone

27. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
MLpersistence
•Prototype(Python/R)
•CreatePipeline
•LoadPipeline(Java/Scala)
–Model.load(“s3n://…”)
•Deployinproduction
DataScienceSoftwareengineering
PersistmodelorPipeline:
model.save(“s3n://…”)

28. ‹#
›
©HortonworksInc.2011–2016.AllRightsReserved
Datascientistsworkwithsoftwareengineer
DataScientistsSoftwareengineers
Exploredata
Createpipeline
Findbestparams
Savemodel
Loadmodel
Deployinproduction
Scoringon
batch/streamingdata