Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automatic Machine Learning, AutoML


Published on

As the complexity of choosing optimised and task specific steps and ML models is often beyond non-experts, the rapid growth of machine learning applications has created a demand for off-the-shelf machine learning methods that can be used easily and without expert knowledge. We call the resulting research area that targets progressive automation of machine learning AutoML.
Although it focuses on end users without expert knowledge, AutoML also offers new tools to machine learning experts, for example to:
1. Perform architecture search over deep representations
2. Analyse the importance of hyperparameters.

Published in: Engineering
  • Be the first to comment

Automatic Machine Learning, AutoML

  1. 1. Automatic Machine Learning By: Himadri Mishra, 13074014
  2. 2. Overview: What is Machine Learning? ● Subfield of computer science ● Evolved from the study of pattern recognition and computational learning theory in artificial intelligence ● Gives computers the ability to learn without being explicitly programmed ● Explores the study and construction of algorithms that can learn from and make predictions on data
  3. 3. Basic Flow of Machine Learning
  4. 4. Overview: Why Machine Learning? ● Some tasks are difficult to define algorithmically. Example: Learning to recognize objects. ● High-value predictions that can guide better decisions and smart actions in real time without human intervention ● Machine learning as a technology that helps analyze these large chunks of big data,
  5. 5. ● Research area that targets progressive automation of machine learning ● Also known as AutoML ● Focuses on end users without expert knowledge ● Offers new tools to Machine Learning experts. ○ Perform architecture search over deep representations ○ Analyse the importance of hyperparameters ○ Development of flexible software packages that can be instantiated automatically in a data-driven way ● Follows the paradigm of Programming by Optimization (PbO) What is Automatic Machine Learning?
  6. 6. Examples of AutoML ● AutoWEKA: Approach for the simultaneous selection of a machine learning algorithm and its hyperparameters ● Deep Neural Networks: notoriously dependent on their hyperparameters, and modern optimizers have achieved better results in setting them than humans (Bergstra et al, Snoek et al). ● Making a science of model search: a complex computer vision architecture could automatically be instantiated to yield state-of-the-art results on 3 different tasks: face matching, face identification, and object recognition.
  7. 7. Methods of AutoML ● Bayesian optimization ● Regression models for structured data and big data ● Meta learning ● Transfer learning ● Combinatorial optimization.
  8. 8. An AutoML Framework
  9. 9. Modules of AutoML Framework, unraveled ● Data Pre-Processing ● Problem Identification and Data Splitting ● Feature Engineering ● Feature Stacking ● Application of various models to data ● Decomposition ● Feature Selection ● Model selection and HyperParameter tuning ● Evaluation of Model
  10. 10. Data Pre-Processing
  11. 11. ● Tabular data is most common way of representing data in machine learning or data mining ● Data must be converted to a tabular form
  12. 12. Problem Identification and Data Splitting
  13. 13. ● Single column, binary values (Binary Classification) ● Single column, real values (Regression problem) ● Multiple column, binary values (Multi-Class Classification) ● Multiple column, real values (Multiple target Regression problem) ● Multilabel Classification Types of Labels
  14. 14. ● Stratified KFold splitting for Classification ● Normal KFold split for regression
  15. 15. Feature Engineering
  16. 16. ● Numerical Variables ○ No Processing Required ● Categorical Variables ○ Label Encoders ○ One Hot Encoders ● Text Variables ○ Count Vectorize ○ TF-IDF vectorize Types of Variables
  17. 17. Feature Stacking
  18. 18. ● Two Kinds of Stacking ○ Model Stacking ■ An Ensemble Approach ■ Combines the power of diverse models into single ○ Feature Stacking ■ Different features after processing, gets combined ● Our Stacker Module is a feature stacker
  19. 19. Application of models and Decomposition
  20. 20. ● We should go for Ensemble tree based models: ○ Random Forest Regressor/Classifier ○ Extra Trees Regressor/Classifier ○ Gradient Boosting Machine Regressor/Classifier ● Can’t apply linear models without Normalization ○ For dense features Standard Scaler Normalization ○ For Sparse Features Normalize without scaling about mean, only to unit variance ● If the above steps give a “good” model, we can go for optimization of hyperparameters module, else continue
  21. 21. ● For High dimensional data, PCA is used to decompose ● For images start with 10-15 components and increase it as long as results improve ● For other kind of data, start with 50-60 components ● For Text Data, we use Singular Value Decomposition after converting text to sparse matrix
  22. 22. Feature Selection
  23. 23. ● Greedy Forward Selection ○ Selecting best features iteratively ○ Selecting features based on coefficients of model ● Greedy backward elimination ● Use GBM for normal features and Random Forest for Sparse features for feature evaluation
  24. 24. Model selection and HyperParameter tuning
  25. 25. ● Most important and fundamental process of Machine Learning
  26. 26. ● Classification: ○ Random Forest ○ GBM ○ Logistic Regression ○ Naive Bayes ○ Support Vector Machines ○ k-Nearest Neighbors ● Regression ○ Random Forest ○ GBM ○ Linear Regression ○ Ridge ○ Lasso ○ SVR Choice of Model and Hyperparameters
  27. 27. Evaluation of Model
  28. 28. Saving all Transformations on Train Data for reuse
  29. 29. Re-Use of saved transformations for Evaluation on validation set
  30. 30. Current Research
  31. 31. Automatic Architecture selection for Neural Network
  32. 32. Automatically Tuned Neural Network ● Auto-Net is a system that automatically configures neural networks ● Achieved the best performance on two datasets in the human expert track of the recent ChaLearn AutoML Challenge ● Works by tuning: ○ layer-independent network hyperparameters ○ per-layer hyperparameters ● Auto-Net submission reached an AUC score of 90%, while the best human competitor (Ideal Intel Analytics) only reached 80% ● first time an automatically-constructed neural network won a competition dataset
  33. 33. Conclusion
  34. 34. ● Machine learning (ML) has achieved considerable successes in recent years and an ever-growing number of disciplines rely on it. ● However, its success crucially relies on human machine learning experts to perform various tasks manually ● The rapid growth of machine learning applications has created a demand for off-the-shelf machine learning methods that can be used easily and without expert knowledge ● Auto-ML is an open research topic and will be very soon challenging the state of the Art results in various domains
  35. 35. Thank You