This document provides an overview of machine learning competitions on Kaggle and tips for participating. It begins with introducing the types of prediction tasks, including classification, regression, and recommendations. It then discusses important considerations like evaluation metrics, data size, and motivation for competing. The rest of the document offers advice on data preprocessing, feature engineering, model selection, ensembling techniques, and learning from other competitors. The overall goal is to understand the machine learning process and get better results by applying diverse models and proper validation.

1. MLDM MONDAY
Chih-Ming

2. About ME
CM 志明
Ph.D Student in TIGP-SNHCC
Research Assistant at AS CITI
Research Intern at KKBOX
Advisor: Prof. Ming-Feng Tsai (蔡銘峰)
Advisor: Dr. Eric Yang (楊弈軒)
• CLIP Lab
• MAC Lab
Research, Machine Learning team
https://about.me/chewme

3. 3 http://kaggletw.azurewebsites.net/

4. 台灣 Kaggle 交流區
https://www.facebook.com/groups/kaggletw/

5. 5

6. 6

7. 7

8. 8

9. First Thing First
• The type of prediction task
- classification? regression? top-N recommendations?
• Evaluation Metric
- AUC, MAE, RMSE, Log-loss, MAP@N, …
• Why Compete?
- For fun
- For learning
- For networking

10. The Prediction Task
Binary
Classification
Multi-label
Classification
Regression
Recommendations

11. Evaluation Metric
https://www.kaggle.com/wiki/Metrics

12. Why Compete?
• For Fun: Competing with others like running or racing
• For Learning: Improving your abilities
• What's Your Motivation?

13. Other Considerations …
• Data Size
- 10MB? 10GB? >100GB?
- no $$ to pay AWS
• Need GPU Power?
- no $$ to pay AWS
• Good Prize?
- $$$$$$$$$$$$

14. Check the Provided Data
• The Distribution of Train/Test Data
- random splitting
- split by time
- split by Ids
• Available Features
- categorical, numerical
- text
- image, audio
- time
- sparse, dense

15. Cross Validation (1) Train
Validation
TRAIN TRAIN
TRAIN TRAIN
TRAIN TRAIN
Round 1:
Round 2:
Round 3:

16. Cross Validation (2) Train
Validation
TRAIN
TRAIN
TRAIN
Test
Round 1:
Round 2:
Round 3:

17. Hold A Proper Validation
• Random Splitting
• Split by Time
• Split by Id
Train
Validation
Test
7 DAYS7 DAYS
5/20 5/275/13
or

18. Data Cleaning / Preprocessing
• Missing Values
- drop the missing data
- replace them by certain statistical values
- label them as the missing value
• Outlier Detection
- https://en.wikipedia.org/wiki/Outlier
• Redundant Features
- remove them usually
mean /
median /
mode /
clustering /
modelling methods

19. Categorical Features
• One-hot Encoding
• Clustering Group
Mayday 1 0 0 0
Sodagreen 0 1 0 0
SEKAI_NO_OWARI 0 0 1 0
The_Beatles 0 0 0 1
Mayday 1 0 0
Sodagreen 1 0 0
SEKAI_NO_OWARI 0 1 0
The_Beatles 0 0 1
Language
Id

20. Categorical Features
• Col-hot Encoding
• Count-hot Encoding
• Likelihood Encoding
• …
T1 T2 T3
T1
T2U
T3
23
1
6
23 1 6
1 0 1
count
binary
probability 23/30 1/30 6/30

21. Categorical Features (2)
• Latent Representations
- Principal Component Analysis (PCA)
- Linear Discriminant Analysis (LDA)
- Laplacian Eigenmaps (LE)
- Locally linear embedding (LLE)
- Low-Rank Approximation / Latent Factorization
- Latent Topic Model
reduce the computation cost alleviating the overfitting issue
finding out the meaningful components remove the noise
https://en.wikipedia.org/wiki/Dimensionality_reduction

22. Categorical vs. Numerical
• Ordinal Categories
HATE DON’T MIND LIKE LOVE
0 1 2 3
0
2
4
6
8
HATE DON'T MIND LIKE LOVE
exp(value)

23. Numerical Features
• Standardization / Normalization
• Rescaling
• Transform the Distribution
- logarithmic transformation
- tf-idf like transformation
• Binning / Sampling
https://en.wikipedia.org/wiki/Feature_scaling
required by
many ML algorithms
https://en.wikipedia.org/wiki/Data_transformation_(statistics)

24. Other Features
• Text-based
- Natural Language Processing
• Image-based, Audio-based
- Image/Signal Processing
• Time-based
- Time Series
Domain Knowledge is Important

25. Example (1)
• Text-based
- Vector Space Model - Word Embeddings
https://en.wikipedia.org/wiki/Vector_space_model
MAN
WOMAN
KING
QUEEN
need stemming? lemmatization?

26. Example (2)
Text
服務好、環境整潔 …
服務⼈人員笑容溫暖...
今天點了商業午餐...
segmentation
[服務] [好] [環境] [整潔]
[服務] [⼈人員] [笑容] [溫暖]
[今天] [點了] [商業午餐]
服務:1 好:1 環境:1 整潔:2
服務:1 笑容:1 溫暖:2
商業午餐:1filtering
Word
Embeddings?
dummy
variables
服務:2 好:1 環境:1 整潔:4
服務:2 笑容:1 溫暖:1
商業午餐:0.8
Advanced
Weighting?

27. Example (3)
• Image-based
- SIFT - Convolutional NN
https://en.wikipedia.org/wiki/Scale-invariant_feature_transform
https://en.wikipedia.org/wiki/Convolutional_neural_network

28. Realize the Meaning Behind the Observed Features
• 2017/05/20 08:00
• Taipei
Holiday? Weekday?
Day? Night?
Asia
Mandarin

29. ML Libraries
• sci-kit learn
• xgboost, lightgbm, …
• vowpal wabbit
• libsvm, liblinear, libfm, libffm, …
• tensorflow, keras, h2o, caffe, mxnet, …
• …

30. Understand the Pros and Cons
• Linear Model
- simple, fast and easy to tune
- occupy low memory
- non-complex
• Nearest Neighbours
- depends on the prediction task and the data distributions

31. Understand the Pros and Cons (2)
• Random Forest
- work very well in many competitions
- fast and easy to tune
- memory hungry
• SVM
- strong theoretical guarantees
- good to prevent from overfit
- slow and memory heavy
- usually needs grid-search on hyper parameters

32. Understand the Pros and Cons (3)
• There are too many details …
• Find some online courses or ML books
• The Elements of Statistical Learning
• Machine Learning, A Probabilistic Perspective
• Programming Collective Intelligence
• Information Science and Statistics
• Pattern Recognition and Machine Learning
• …

33. Exploratory Data Analysis (EDA)
• Statistics Helps
- min, max, variance, mode, …
• Data Visualization Helps

34. Model Ensembling
• Voting
• Averaging
• Bagging
• Boosting
• Blending
• Stacking
1 2 3 4
Avg.
Avg.

35. Model Ensembling
• Voting
• Averaging
• Bagging
• Boosting
• Blending
• Stacking
1 2 3 4
ENSEMBLE
MODEL

36. Model Ensembling
• Voting
• Averaging
• Bagging
• Boosting
• Blending
• Stacking
1 2 3 4 new feature
Avg.

37. Other Tricks
• Data Leakage
• Magic/Lucky Parameters

38. Overall - To Get into Top
• Correct Validation
• Good Feature Extractions
• Diverse Model
• Proper Ensembling

39. https://www.slideshare.net/NathanielShimoni/starting-data-science-with-kagglecom?qid=8f0c66f9-43ba-4646-8a05-
d03bf30b2eeb&v=&b=&from_search=9

40. http://blog.kaggle.com/2016/07/21/approaching-almost-any-machine-learning-problem-abhishek-thakur/

41. Learning from Others/Winners
• Discussion Forum
• Kernels
• Winner Solutions

42. Learning from Others/Winners
http://blog.kaggle.com/

43. ANY QUESTION?
changecandy at gmail