Distinguish Pop from Heavy Metal using Apache Spark MLlib
•
1 like•686 views
Machine learning may be overhyped nowadays, but there is still a strong belief that this area is exclusively for data scientists with a deep mathematical background who leverage the Python (scikit-learn, Theano, TensorFlow, etc.) or R ecosystems and use specific tools like R Studio, Matlab, or Octave. Obviously, there is some truth to this statement, but Java engineers can also take the best of the machine-learning world from an applied perspective by using our native language and familiar frameworks like Apache Spark. Taras Matyashovsky explains how to use Apache Spark MLlib to build a supervised learning NLP pipeline to distinguish pop music from heavy metal—and have fun in the process. Along the way, Taras offers an overview of the simplest machine-learning tasks and algorithms, like regression and classification. Source code: https://github.com/tmatyashovsky/spark-ml-samples Design by Yarko Filevych: http://filevych.com/
Recommended
Recommended
More Related Content
More from Taras Matyashovsky
More from Taras Matyashovsky (8)
Recently uploaded
Recently uploaded (20)
Distinguish Pop from Heavy Metal using Apache Spark MLlib
- 1. using Apache Spark MLlib #javaone
- 3. I am not a data science engineer 3
- 4. 4
- 6. “I'm a rolling thunder, a pouring rain I'm comin' on like a hurricane My lightning's flashing across the sky You're only young but you're gonna die I won't take no prisoners, won't spare no lives Nobody's putting up a fight I got my bell, I'm gonna take you to hell I'm gonna get you, Satan get you” https://github.com/tmatyashovsky/spark-ml-samples 6
- 7. “I'm a rolling thunder, a pouring rain I'm comin' on like a hurricane My lightning's flashing across the sky You're only young but you're gonna die I won't take no prisoners, won't spare no lives Nobody's putting up a fight I got my bell, I'm gonna take you to hell I'm gonna get you, Satan get you” https://github.com/tmatyashovsky/spark-ml-samples 7
- 8. 8
- 9. Look for particular words like “fear”, “fight”, “kill”, “devil”, ”death”, etc.? Count length of a verse? Count unique words in a verse? 9
- 10. 10
- 11. 15 11
- 12. is the study of computer algorithms that improve automatically through experience 12
- 14. 14
- 15. Date & time Conference name Speaker Talk name Track Duration Type Overall impression Overall rating Number of slides Time spent on live coding Number of jokes Etc. 15
- 16. Learning algorithms Hypotheses: Сost function: Features: Target variable: Training example: Training set: 16
- 18. Number of jokes during a talk Speaker’s rating 18
- 19. 19
- 20. 20
- 21. 21
- 22. 22
- 23. 23
- 24. 24
- 25. Positive Negative Impression Number of jokes during a talk 25
- 26. 26
- 27. 27
- 28. 28
- 29. 29
- 30. 30
- 31. 31
- 32. Collect data set of lyrics: Abba, Ace of base, Backstreet Boys, Britney Spears, Christina Aguilera, Madonna, etc. Black Sabbath, In Flames, Iron Maiden, Metallica, Moonspell, Nightwish, Sentenced, etc. Create training set, i.e. label (0|1) + features Train logistic regression (or other classification algorithm) https://github.com/tmatyashovsky/spark-ml-samples 32
- 34. 34
- 36. Produces unique fixed-size dense vectors Captures semantic and morphologic similarity https://code.google.com/archive/p/word2vec/ 36
- 37. Similar scores (cos ~ 1) Opposite scores (cos ~ -1) Unrelated scores (cos ~ 0) http://bionlp-www.utu.fi/wv_demo/ http://blog.christianperone.com/wp-content/uploads/2013/09/cosinesimilarityfq1.png 37
- 38. 38 Verse Cosine Distance baby one more time 0.482028 crazy for you 0.437875 show me the meaning of being lonely 0.258147 highway to hell -0.1120049 kill them all -0.231876 https://github.com/tmatyashovsky/spark-ml-samples
- 43. Training set (66,6%) Test set (33%) K = 3 43
- 44. Training set (66,6%) Test set (33%) K = 3 44
- 45. Training set (33,3%) Test set (33%) Training set (33,3%) K = 3 45
- 46. 46
- 47. Java 47
- 51. Is a library of ML algorithms and utilities designed to run in parallel on Spark cluster 51
- 52. Introduces a few new data types, e.g. vector (dense and sparse), labeled point, rating, etc. Allows to invoke various algorithms on distributed datasets (RDD/Dataset) http://spark.apache.org/docs/latest/mllib-guide.html 52
- 53. http://spark.apache.org/docs/latest/mllib-guide.html Build on top of RDDs Build on top of Datasets spark.mll ib spark.ml 53
- 54. Utilities: linear algebra, statistics, etc. Features extraction, features transforming, etc. Regression Classification Clustering Collaborative filtering, e.g. alternating least squares Dimensionality reduction And many more http://spark.apache.org/docs/latest/mllib-guide.html 54
- 55. ”All” spark.mllib features plus: • Pipelines • Persistence • Model selection and tuning: • Train validation split • K-folds cross validation http://spark.apache.org/docs/latest/ml-guide.html 55
- 56. Raw data Transformer Estimator [parameters] Transformer [parameters] Estimator [parameters] Dataset Dataset Dataset Dataset http://spark.apache.org/docs/latest/ml-pipeline.html Cross Validator [pipeline, evaluator, parameters] Dataset 56
- 57. Using Spark MLlib Pipeline
- 59. I'm a rolling thunder, a pouring rain I'm comin' on like a hurricane My lightning's flashing across the sky You're only young but you're gonna die I won't take no prisoners, won't spare no lives Nobody's putting up a fight I got my bell, I'm gonna take you to hell I'm gonna get you, Satan get you https://github.com/tmatyashovsky/spark-ml-samples 59
- 61. I'm a rolling thunder, a pouring rain I'm comin' on like a hurricane My lightning's flashing across the sky You're only young but you're gonna die I won't take no prisoners, won't spare no lives Nobody's putting up a fight I got my bell, I'm gonna take you to hell I'm gonna get you, Satan get you https://github.com/tmatyashovsky/spark-ml-samples 61
- 63. Im a rolling thunder a pouring rain Im comin on like a hurricane My lightnings flashing across the sky Youre only young but youre gonna die I wont take no prisoners wont spare no lives Nobodys putting up a fight I got my bell Im gonna take you to hell Im gonna get you Satan get you https://github.com/tmatyashovsky/spark-ml-samples 63 1 2 3 4 5 6 7 8
- 64. Lyrics Cleanser Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 64
- 65. im a rolling thunder a pouring rain im comin on like a hurricane My lightnings flashing across the sky youre only young but youre gonna die I wont take no prisoners wont spare no lives nobodys putting up a fight I got my bell im gonna take you to hell im gonna get you satan get you https://github.com/tmatyashovsky/spark-ml-samples 65 1 2 3 4 5 6 7 8
- 66. Lyrics Cleanser Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 66
- 67. im rolling thunder pouring rain im comin like hurricane lightnings flashing across sky youre young youre gonna die wont take prisoners wont spare lives nobodiys putting fight got bell im gonna take hell im gonna get satan get https://github.com/tmatyashovsky/spark-ml-samples 67 1 2 3 4 5 6 7 8
- 68. Lyrics Cleanser Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 68
- 69. 4 im roll thunder pour rain im comin like hurrican lightn flash across sky your young your gonna die wont take prison wont spare live nobodi put fight got bell im gonna take hell im gonna get satan get https://github.com/tmatyashovsky/spark-ml-samples 69 1 2 3 4 5 6 7 8 verse1 verse2
- 70. 8 im roll thunder pour rain im comin like hurrican Light n flash across sky your young your gonna die wont take prison wont spare live nobodi put fight got bell im gonna take hell im gonna get satan get https://github.com/tmatyashovsky/spark-ml-samples 70 1 2 3 4 5 6 7 8 verse1
- 71. Lyrics Cleanser Word2Vec [Vector size] Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 71
- 74. Lyrics Cleanser Word2Vec [Vector size] Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Logistic Regression [Max iterations, Reg parameter] Dataset Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 74
- 76. Lyrics Cleanser Word2Vec [Vector size] Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Logistic Regression [Max iterations, Reg parameter] Dataset Dataset Cross Validator Model Dataset https://github.com/tmatyashovsky/spark-ml-samples 76
- 78. Lyrics Cleanser Word2Vec [Vector size] Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Logistic Regression [Max iterations, Reg parameter] Dataset Dataset Cross Validator Model Dataset https://github.com/tmatyashovsky/spark-ml-samples 78
- 79. 79
- 81. • Other feature extractors: • Term Frequency – Inverse Document Frequency (TD-IDF), Token counts (TF), etc. • Other classification algorithms: • Naive Bayes, Random Forest, Support Vector Machines (SVM), etc. http://spark.apache.org/docs/latest/ml-guide.html 81
- 89. Lyrics Cleanser Word2Vec [Vector size] Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Logistic Regression [Max iterations, Reg parameter] Dataset Dataset Cross Validator Model Dataset https://github.com/tmatyashovsky/spark-ml-samples 89
- 90. Lyrics Cleanser Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Naive Bayes Dataset Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 90 Hashing TF [Num Features] IDF [Min Doc Freq] Dataset Cross Validator Model
- 91. 91
- 92. 92
- 93. 93 ML is not as complex as it seems from an applied perspective Existing libraries and frameworks reduce a lot of tedious work For instance, Spark MLlib can help to build nice ML pipelines
- 94. Design by 94
- 95. https://www.quora.com/What-is-the-difference-between-supervised-and-unsupervised-learning-algorithms Learning Spark, by Holden Karau, Andy Konwinski, Patrick Wendell and Matei Zaharia https://databricks.com/blog/2015/01/07/ml-pipelines-a-new-high-level-api-for-mllib.html https://databricks.com/blog/2016/05/31/apache-spark-2-0-preview-machine-learning-model-persistence.html https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research https://www.kaggle.com/c/dogs-vs-cats/ http://yann.lecun.com/exdb/mnist/ http://www.bcl.hamilton.ie/~barak/teach/F98/ECE547/hw1/index.html http://www.slideshare.net/jeykottalam/pipelines-ampcamp https://github.com/master/spark-stemming https://databricks.com/blog/2016/04/01/unreasonable-effectiveness-of-deep-learning-on-apache-spark.html http://www.degeneratestate.org/posts/2016/Apr/20/heavy-metal-and-natural-language-processing-part-1/ https://spark.apache.org/docs/2.0.0/api/java/org/apache/spark/sql/functions.html https://www.quora.com/What-is-the-difference-between-supervised-and-unsupervised-learning-algorithms http://www.slideshare.net/liweiyang5/spark-mllib-training-material https://databricks.com/blog/2016/01/25/deep-learning-with-apache-spark-and-tensorflow.htm http://www.slideshare.net/databricks/combining-machine-learning-frameworks-with-apache-spark l https://databricks.com/blog/2015/10/20/audience-modeling-with-apache-spark-ml-pipelines.html https://github.com/deeplearning4j/deeplearning4j http://deeplearning4j.org/spark http://mlwiki.org/index.php/Overfitting http://bionlp-www.utu.fi/wv_demo/ https://quomodocumque.wordpress.com/2016/01/15/messing-around-with-word2vec/ 95
Editor's Notes
- Score of the speaker based on xxx.
- Quantity of jokes used. Liked or not liked the speaker.
- Bag of words – a single word is a one hot encoding vector with the size of the dictionary. As a result – a lot of sparse vectors.
- Behind the scenes - a two-layer neural net that processes text. Captures semantic and morphologic similarity so similar words are close in the vector space Similar words would be clustered together in the high dimensional sphere.
- If two words are very close to synonymous, you’d expect them to show up in similar contexts, and indeed synonymous words tend to be close. For two completely random words, the similarity is pretty close to 0. On an opposite side there is not an antonym, but usually just a noise. Used Google News Negative 300.
- My corpus - 8316 words
- Let’s finally go to the implementation using a library or framework that is going to help us to avoid tedious transformations and provide algorithms as well as feature extractors out-of-the-box.