Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kaggle and data science

329 views

Published on

How is Kaggle different from real data science? The talk was given by Kaggle Grandmaster Sergey Yurgenson

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Kaggle and data science

  1. 1. © DataRobot, Inc. All rights reserved. Kaggle and Data Science Japan, 2018
  2. 2. Sergey Yurgenson Director, Advanced Data Science Services Kaggle Grandmaster © DataRobot, Inc. All rights reserved.
  3. 3. © DataRobot, Inc. All rights reserved. Kaggle ● Kaggle is a platform for data science competitions ● It was created by Anthony Goldbloom in 2010 in Australia and then moved to San Francisco ● In March of 2017 it was acquired by Google ● Right now many other start-up are trying to replicate the same idea, but Kaggle is still the most known in data science community name ● As of now Kaggle hosted more than 280 competitions and has more than 1 million members from more than 190 countries
  4. 4. © DataRobot, Inc. All rights reserved. Kaggle competitions ● Most of Kaggle competitions are predictive modeling competition ● Participants are provided with training data to train their models and test data with unknown targets ● Participants need to calculate predictions for test data and submit those predictions to Kaggle platform. ● Accuracy of predictions is evaluated using predefined objective metric and that result is provided back to participants. ● Model performance of all participants is publicly available and participants can compare quality of their models with models of other participants ● Many competitions have monetary prizes for top finishers
  5. 5. © DataRobot, Inc. All rights reserved. Kaggle competitions
  6. 6. © DataRobot, Inc. All rights reserved. Kaggle ranking ● Based on competitions performance Kaggle ranks members using points and awards titles for top finishing in competitions ● For example to get title of master member needs to earn one gold medal and two silver medal. For competitions with 1000 participants it means to finish once in top 10 places and twice in top 50.
  7. 7. © DataRobot, Inc. All rights reserved. Kaggle ranking
  8. 8. © DataRobot, Inc. All rights reserved. Kaggle and Data Science
  9. 9. © DataRobot, Inc. All rights reserved. Why do you dislike Kaggle ? ● Kaggle competition does not have much in common with real Data Science ○ The problems are already well formulated with metrics predefined. In an industry setting there is ambiguity, and knowing what to solve is one of the key steps towards a solution. ○ Data is most cases is already provided and is relatively clean. ○ The goal is more leaderboard driven rather than understanding driven. Winning a competition versus why an approach works is a top priority. Results may not be trustworthy. ○ There are chances of overfitting to test data with repeated submissions. ○ In most cases the solution is an ensemble of algorithms and not “productionizable”. https://www.quora.com/Why-do-you-dislike-Kaggle
  10. 10. © DataRobot, Inc. All rights reserved. True or False ? ● “The problems are already well formulated with metrics predefined. In an industry setting there is ambiguity, and knowing what to solve is one of the key steps towards a solution.” https://www.quora.com/Why-do-you-dislike-Kaggle
  11. 11. © DataRobot, Inc. All rights reserved. Problem is well formulated Mostly True , however... ● Need for criteria is inherited property of any competition. ● In real world not all data scientists are free to select and reformulate the problem. Many problems are already defined with assigned specific success criteria. ● We learn many subjects and skills by solving provided predefined problems, doing predefined exercises. We learn math by solving problems from textbooks, we learn physics by solving problems from textbooks. Problems already formulated. By solving problems we also learn how to formulate problems, what is suitable approach in particular data science situation. ● We also have to admit that evaluating business value of solving the problem is completely out of scope of Kaggle competitions. While business value analysis and problem prioritization is important part of many real life data science projects.
  12. 12. © DataRobot, Inc. All rights reserved. True or False ? ● “Data is most cases is already provided and is relatively clean.” https://www.quora.com/Why-do-you-dislike-Kaggle
  13. 13. © DataRobot, Inc. All rights reserved. Data is clean Half true ● In many competitions datasets are ○ Very big ○ Have multiple tables ○ Some records are duplicated and mislabeled ○ Contain combination of structured data and unstructured data ● Some competitions encourage search for additional sources of data ● Many data leaks ● Often features names and meaning are not provided making problem even more difficult than in real world ● Data may be intentionally distorted to conform to data privacy laws
  14. 14. © DataRobot, Inc. All rights reserved. Data is clean ● Complex data structure ● Big datasets ● No meaningful feature names
  15. 15. © DataRobot, Inc. All rights reserved. Data is clean ● Kaggle competitions teach unique data manipulation skills: ○ Dealing with data with hardware limitations : efficient code, smart sampling, clever encoding… ○ Using EDA to uncover meaning of data without relying on labels or other provided information ○ Data leaks discovering based on the data analysis
  16. 16. © DataRobot, Inc. All rights reserved. True or False ? ● The goal is more leaderboard driven rather than understanding driven. Winning a competition versus why an approach works is a top priority. Results may not be trustworthy. https://www.quora.com/Why-do-you-dislike-Kaggle
  17. 17. © DataRobot, Inc. All rights reserved. No understanding True but maybe not that important ● Assumes that model we can not understand is less valuable than model we can understand ○ Model is not necessarily used for knowledge discovery ○ In real life we often use something and rely on something we do not completely understand ○ If something that we do not understand can not be trustworthy then how we ever trust other people? ○ Even complex machine learning model may provide simplification of even more complex real system
  18. 18. © DataRobot, Inc. All rights reserved. No understanding ● Ignores all new research of model interpretability ○ Feature importance ○ Reason codes ○ Partial dependence plots ○ Surrogate models ○ Neuron activation visualization ○ ... ● Those methods allow us to analyze and understand behaviour of models as complicated as GBM and Neural Networks
  19. 19. © DataRobot, Inc. All rights reserved. No understanding ?
  20. 20. © DataRobot, Inc. All rights reserved. True or False ? ● There are chances of overfitting to test data with repeated submissions. https://www.quora.com/Why-do-you-dislike-Kaggle
  21. 21. © DataRobot, Inc. All rights reserved. Overfitting False ● Complete misunderstanding of how Kaggle works ○ Test data in Kaggle competition is split into two parts - public and private ○ During competition models are evaluated only on public part of the test set ○ Final results are based only on private part of the test dataset ○ Thus final model evaluation is based on completely new data ● One of first lessons all competitions participants learn very fast ○ Do not overfit leaderboard. ○ Create training/validation partition which reflect as much as possible test data including seasonality effects and data drift
  22. 22. © DataRobot, Inc. All rights reserved. True or False ? ● In most cases the solution is an ensemble of algorithms and not “productionizable”. https://www.quora.com/Why-do-you-dislike-Kaggle
  23. 23. © DataRobot, Inc. All rights reserved. Difficult to put in production Half True, half false ● Yes, in most cases top models are complicated ensembles ● Difficult to put in production if one does it one-by-one for each model separately ● Easy if one uses appropriately developed platform that can handle many models and blenders
  24. 24. © DataRobot, Inc. All rights reserved. True or False ? ● Sometimes, a 0.01 difference in AUC can be the difference between 1st place and 294th place (out of 626) . Those marginal gains take significant time and effort that may not be worthwhile in the face of other projects and priorities https://www.quora.com/How-similar-are-Kaggle-competitions-to-what-data-scientists-do
  25. 25. © DataRobot, Inc. All rights reserved. Marginal gain is not valuable Not always true ● Often we ourselves advise clients on balance between time spent and model performance ● However in investment world 0.01 AUC difference means difference in millions of dollars of gain or loss ● Competition aspect of the data science problem with small margins drives innovation ○ New preprocessing steps ○ New feature engineering ideas ○ Continues testing of new algorithms and implementations (GBM - XGboost - LightGBM - CatBoost)
  26. 26. © DataRobot, Inc. All rights reserved. Kaggle and Data Science ● “Kaggle competitions cover a decent amount of what a data scientist does. The two big missing pieces are: ○ 1. taking a business problem and specifying it as a data science problem (which includes pulling the data and structuring it so that it addresses that business problem). ○ 2. putting models into production.” Anthony Goldbloom
  27. 27. © DataRobot, Inc. All rights reserved. Kaggle and Data Science ● Kaggle is a competition ● “Real” Data Science is ... also competition
  28. 28. © DataRobot, Inc. All rights reserved. Kaggle to “real life” Data Science ● DataRobot - created by top Kagglers Owen Zhang Product Advisor Highest: #1 Xavier Conort Chief Data Scientist Highest: 1st Sergey Yurgenson Director- AI Services Highest: 1st Jeremy Achin CEO & Co-Founder Highest: 20th Tom de Godoy CTO & Co-Founder Highest: 20th Amanda Schierz Data Scientist Highest: 24 DataRobot automatically replicates the steps seasoned data scientists take. This allows non-technical business users to create accurate predictive models and data scientists to add to their existing tool set.
  29. 29. © DataRobot, Inc. All rights reserved. Kaggle and Data Science

×