Data Science Competition
•Download as PPTX, PDF•
15 likes•2,320 views
Presented at the 27th Annual KSEA South-Western Regional Conference (KSEA-SWRC 2017) on February 25, 2017
Recommended
Recommended
More Related Content
What's hot
What's hot (14)
Viewers also liked
Viewers also liked (20)
Similar to Data Science Competition
Similar to Data Science Competition (20)
Recently uploaded
Recently uploaded (20)
Data Science Competition
- 1. Data Science Competition 2. 25. 2017 The 27th Annual KSEA South-Western Regional Conference Jeong-Yoon Lee, Ph.D.
- 2. Chief Data Scientist, Conversion Logic Ph.D. in Computer Science, USC M.S. in Electrical Engineering, USC B.S. in Electrical Engineering, SNU KDD Cup Winner 2012 & 2015 Top 10, Kaggle 2015 Jeong-Yoon Lee, Ph.D.
- 3. Why Data Science Competition
- 4. Why Compete • For fun • For experience • For learning • For networking 4
- 5. Fun • Competing with others • Incremental improvement 5
- 6. Experience 6
- 7. Learning 7
- 8. Learning 8
- 9. Networking 9
- 10. 10
- 12. Data Science Competitions Since 1997 2006 - 2009 Since 2010
- 13. Competition Structure Training Data Test Data Feature Label Provided Submission Public LB Score Private LB Score
- 14. Kaggle • 250+ competitions since 2010 • 500K+ users • 50K+ competitors • $3MM+ prize paid out
- 15. Kaggle
- 16. Kaggle
- 18. Misconceptions on Competitions • No ETL • No EDA • Not worth it • Not for production 18
- 19. No ETL? - Deloitte Western Australia Rental Prices 19
- 20. No ETL? - Outbrain Click Prediction 20 2B page views. 16.9MM clicks. 700MM users. 560 sites
- 21. No ETL? - YouTube-8M Video Understanding Challenge 21 1.7TB feature-level data. 31GB video-level data.
- 22. No ETL? 22
- 23. No EDA? • Most of competitions provide actual labels - typical EDA • Anonymized data - more creative EDA o People decode age, states, time intervals, income, etc. 23
- 24. No EDA? • Anonymized data - more creative EDA 24
- 25. Not worth it? • Performance matters • You walk easier when you can run 25
- 26. Not for Production? • Kaggle Kernel o Max execution time:10 minutes o Max file output: 500MB o Memory limit: 8GB 26
- 27. Ensemble Pipeline at Conversion Logic 27
- 28. Best Practices
- 29. Best Practices • Feature Engineering • Algorithms • Cross Validation • Ensemble 29
- 30. Feature Engineering • Numerical - Log, Log(1 + x), Normalization, Binarization • Categorical - One-hot-encode, TF-IDF (text), Weight-of- Evidence • Timeseries - Stats, FFT, MFCC, ERP (EEG) • Numerical/Timeseries to Categorical - RF/GBM* • Dimensionality Reduction - PCA, SVD, Autoencoder * http://www.csie.ntu.edu.tw/~r01922136/kaggle-2014-criteo.pdf 30
- 31. Algorithms Algorithm Tool Note Gradient Boosting Machine XGBoost, LightGBM The most popular algorithm in competitions Random Forests Scikit-Learn, randomForest Used to be popular before GBM Extremely Random Trees Scikit-Learn Neural Networks/ Deep Learning Keras, MXNet, CNTK, Torch Blends well with GBM. Best at image and speech recognition competitions Logistic/Linear Regression Scikit-Learn, Vowpal Wabbit Fastest. Good for ensemble. Support Vector Machine Scikit-Learn FTRL Vowpal Wabbit Competitive solution for CTR estimation competitions Factorization Machine libFM Winning solution for KDD Cup 2012 Field-aware Factorization Machine libFFM Winning solution for CTR estimation competitions (Criteo, Avazu) 31
- 32. Cross Validation Training data are split into five folds where the sample size and dropout rate are preserved (stratified). 32
- 34. Ensemble * for other types of ensemble, see http://mlwave.com/kaggle-ensembling-guide/ 34
- 36. Why Competition • For fun • For experiences • For learning • For networking 36
- 37. One Last Thing 37 Google: 20K applications per week Conversion Logic: 200 applications per week
- 38. Thank You
Editor's Notes
- I am Jeong-Yoon Lee, Chief Data Scientist at Conversion Logic. I am going to tell you little bit about our attribution approach.
- states, age, time interval, weekday,
- states, age, time interval, weekday,
- Training data are split into five folds while the sample size and dropout rate are preserved across folds. For validation, each of single and ensemble models is trained five times. Each time, one fold is held out and the remain- ing four folds are used for training. Then, predictions for the hold-out folds are combined and form the model’s CV pre- diction. CV predictions are used in AUC score calculation and/or as inputs in ensemble model training. For test, each of single and ensemble models is retrained with whole training data. Then predictions for test data are used for submission and/or as inputs in ensemble model prediction.
- For validation, each of single and ensemble models is trained five times. Each time, one fold is held out and the remain- ing four folds are used for training. Then, predictions for the hold-out folds are combined and form the model’s CV pre- diction. CV predictions are used in AUC score calculation and/or as inputs in ensemble model training. For test, each of single and ensemble models is retrained with whole training data. Then predictions for test data are used for submission and/or as inputs in ensemble model prediction.
- Stage-I Ensemble: We trained 15 stage-I ensemble classifiers with different subsets of CV predictions of 64 individual classifiers. Stage-II Ensemble: We trained 2 stage-II ensemble classifiers with different subsets of CV predictions of 15 stage-I ensemble classifiers. Stage-III Ensemble: We trained a stage-III ensemble classifier with CV predictions of 5 classifiers: 1 stage-II ensemble, 3 stage-I ensemble, and 1 individual classifiers