Wendy Kan, Data Scientist at Kaggle, explains how Kaggle competitions are created at H2O.ai, 5/19/15.

1. stories behind kaggle competitions
wendy kan, data scientist
wendy@kaggle.com
@wendykan
5/19/2015 @

2. kaggle runs public machine learning
competitions

3. we worked with clients/hosts on various
types of problems and data of different sizes

4. my job as a data scientist at kaggle

5. “data science is not just
kaggle competitions”
whyyyy???

6. machine learning processes
● Business Problem
● Collect Data
● Transform Data
● Dataset Splitting
● Evaluation Metric
● Feature Extraction
● Feature Selection
● Model Training
● Model Ensembling
● Methodology Selection
● Production System
● Ongoing Optimization

7. not every problem can be
turned into a kaggle
competition

9. size matters! where bigger is
better (most of the time)

10. data cleaning/formatting:
● easy to make a quick submission
● boosts participation
● (too) clean data kills creativity

11. data privacy/anonymization

12. metric: how do you measure
success?
● Classification - AUC/ Logarithmic Loss/Accuracy
● Regression - RMSE/MAE
● Ranking - MAP/NDCG
● Other / Custom
https://www.kaggle.com/wiki/Metrics

13. the design of a competition shapes how
people are going to solve a problem

14. Splitting dataset
● training/test
● public/private

15. Time series data

16. data leakage
“Deemed ‘one of the top ten data mining mistakes’, leakage is
essentially the introduction of information about the data mining target,
which should not be legitimately available to mine from”
“the concept of identifying and harnessing leakage has been openly
addressed as one of three key aspects for winning data mining
competitions”
“Leakage in Data Mining: formulation, detection, and avoidance” S Kaufman et al

17. do you have thousands of
people reviewing your
performance at work 24/7?
I do.

18. 1. people make mistakes.
honesty is the best policy.

19. 2. crowdsourcing is powerful.
anything that can go wrong
will go wrong.