Abstract: Are the machines learning on their own? Wait, is Skynet already here? During this session we will tackle an easy Machine Learning problem, show how can it be processed on Spark including data cleaning, normalization and a learning process. Live coding session, but only if the machines don’t rise against us.

1. Machine learning by example
Michał Matłoka @mmatloka

2. –Alan Turing (1950)
“Can machines think?”

3. Turing test
http://www.smbc-comics.com/?id=2999

4. • Make machines
“think” like humans
• Learn from data and
make predictions
What is ML?

5. • Supervised learning
◦ Classification
◦ Regression
• Unsupervised learning
◦ Clustering
◦ Dimensionality Reduction
• Semi-supervised learning
• Reinforcement learning
◦ E.g. AlphaGo
Learning
types

6. SUPERVISED - CLASSIFICATION

7. SUPERVISED - REGRESSION

8. UNSUPERVISED - CLUSTERING

9. UNSUPERVISED -
DIMENSIONALITY REDUCTION

10. SEMI SUPERVISED

11. REINFORCMENT LEARNING
ALPHAGO
https://cdn.arstechnica.net/wp-content/uploads/sites/3/2017/05/GettyImages-688097364-800x549.jpg

12. • Voice recognition
• Fraud analysis
• Face detection
• Ads click-through rate
prediction
• Spam detection
• Shop recommendations
• Photos description
• Self-driving cars
• Healthcare
• ...
Use cases

13. 1. Data gathering
2. Data cleaning & feature
engineering
3. Dataset -> training & test set
4. Learning -> Model
5. Evaluation -> Accuracy
6. New observation -> Prediction
Learning
process

14. EXAMPLE

15. Classify conference
talk abstracts into
tracks

16. • RDD (Resilient Distributed Dataset) - map,
filter, count etc - DataFrame
• Spark SQL
• MLib
• GraphX
• Spark Streaming
• API: Scala, Java, Python, R*

17. Code.
https://github.com/mmatloka/machine-learning-by-example

18. • Bigger data set
• Smarter tokenizer
• Stemming &
lemmatization
• IDF - Inverse Document
Frequency
• K-fold Cross-Validation
• Parameters tuning (grid-
search)
What can be
improved?

19. “I will present Big Data
topics”
“I”, “will”, “present”, “Big”,
“Data”, “topics”
What can be
improved?
(Tokenizer)

20. “I will present Big Data
topics”
“I will”, “will present”,
“present Big”, “Big Data”,
“Data topics”
What can be
improved?
(n-grams)

21. STEMMING & LEMMATIZATION

22. “communities”, community”
“commun”
What can be
improved?
(Porter Stemmer)

23. “communities”, community”
“community”
What can be
improved?
(Lemmatization)

24. TF - Term Frequency
(HashingTF!) - number of
times term occurs in given
document
IDF - Inverse Document
Frequency - occurs many
times in documents set
What can be
improved?
(TF & IDF)

25. What can be
improved?
(K-fold Cross-Validation)

26. val paramGrid = new
ParamGridBuilder()
.addGrid(hashingTF.numFeatures,
Array(256, 512, 1024, 2048, 4096))
.build()
val cv = new CrossValidator()
.setEstimator(pipeline)
.setEvaluator(evaluator)
.setEstimatorParamMaps(paramGrid)
.setNumFolds(5)
val cvModel = cv.fit(trainData)
What can be
improved?
(Grid search)

27. Articles & references
• https://www.csee.umbc.edu/courses/471/papers/turing.pdf
• http://spark.apache.org/
• https://databricks.com/try-databricks
• https://research.googleblog.com/2016/09/introducing-
open-images-dataset.html
• https://arstechnica.co.uk/information-technology/2017/05/
deepmind-alphago-go-ke-jie-china/
• https://www.kaggle.com
• https://github.com/dylanmei/docker-zeppelin
• https://github.com/databricks/spark-corenlp
• https://www.coursera.org/learn/scala-spark-big-data

28. Thank you, Q&A?
@mmatloka
mmatloka
softwaremill.com/blog