As the complexity of choosing optimised and task specific steps and ML models is often beyond non-experts, the rapid growth of machine learning applications has created a demand for off-the-shelf machine learning methods that can be used easily and without expert knowledge. We call the resulting research area that targets progressive automation of machine learning AutoML.
Although it focuses on end users without expert knowledge, AutoML also offers new tools to machine learning experts, for example to:
1. Perform architecture search over deep representations
2. Analyse the importance of hyperparameters.
2. Overview: What is Machine Learning?
● Subfield of computer science
● Evolved from the study of pattern recognition and
computational learning theory in artificial intelligence
● Gives computers the ability to learn without being
explicitly programmed
● Explores the study and construction of algorithms that
can learn from and make predictions on data
4. Overview: Why Machine Learning?
● Some tasks are difficult to define algorithmically.
Example: Learning to recognize objects.
● High-value predictions that can guide better decisions
and smart actions in real time without human intervention
● Machine learning as a technology that helps analyze these
large chunks of big data,
5. ● Research area that targets progressive automation of
machine learning
● Also known as AutoML
● Focuses on end users without expert knowledge
● Offers new tools to Machine Learning experts.
○ Perform architecture search over deep representations
○ Analyse the importance of hyperparameters
○ Development of flexible software packages that can be instantiated
automatically in a data-driven way
● Follows the paradigm of Programming by Optimization (PbO)
What is Automatic Machine Learning?
6. Examples of AutoML
● AutoWEKA: Approach for the simultaneous selection of a machine learning
algorithm and its hyperparameters
● Deep Neural Networks: notoriously dependent on their hyperparameters, and
modern optimizers have achieved better results in setting them than humans
(Bergstra et al, Snoek et al).
● Making a science of model search: a complex computer vision architecture
could automatically be instantiated to yield state-of-the-art results on 3
different tasks: face matching, face identification, and object
recognition.
7. Methods of AutoML
● Bayesian optimization
● Regression models for structured data and big data
● Meta learning
● Transfer learning
● Combinatorial optimization.
10. Modules of AutoML Framework, unraveled
● Data Pre-Processing
● Problem Identification and Data Splitting
● Feature Engineering
● Feature Stacking
● Application of various models to data
● Decomposition
● Feature Selection
● Model selection and HyperParameter tuning
● Evaluation of Model
19. ● Two Kinds of Stacking
○ Model Stacking
■ An Ensemble Approach
■ Combines the power of diverse models into single
○ Feature Stacking
■ Different features after processing, gets combined
● Our Stacker Module is a feature stacker
21. ● We should go for Ensemble tree based models:
○ Random Forest Regressor/Classifier
○ Extra Trees Regressor/Classifier
○ Gradient Boosting Machine Regressor/Classifier
● Can’t apply linear models without Normalization
○ For dense features Standard Scaler Normalization
○ For Sparse Features Normalize without scaling about mean, only to
unit variance
● If the above steps give a “good” model, we can go for
optimization of hyperparameters module, else continue
22. ● For High dimensional data, PCA is used to decompose
● For images start with 10-15 components and increase it as
long as results improve
● For other kind of data, start with 50-60 components
● For Text Data, we use Singular Value Decomposition after
converting text to sparse matrix
24. ● Greedy Forward Selection
○ Selecting best features iteratively
○ Selecting features based on coefficients of model
● Greedy backward elimination
● Use GBM for normal features and Random Forest for Sparse
features for feature evaluation
34. Automatically Tuned Neural Network
● Auto-Net is a system that automatically configures neural networks
● Achieved the best performance on two datasets in the human expert track of
the recent ChaLearn AutoML Challenge
● Works by tuning:
○ layer-independent network hyperparameters
○ per-layer hyperparameters
● Auto-Net submission reached an AUC score of 90%, while the best human
competitor (Ideal Intel Analytics) only reached 80%
● first time an automatically-constructed neural network won a competition
dataset
36. ● Machine learning (ML) has achieved considerable successes
in recent years and an ever-growing number of disciplines
rely on it.
● However, its success crucially relies on human machine
learning experts to perform various tasks manually
● The rapid growth of machine learning applications has
created a demand for off-the-shelf machine learning
methods that can be used easily and without expert
knowledge
● Auto-ML is an open research topic and will be very soon
challenging the state of the Art results in various
domains
