Automated machine learning (automated ML) automates feature engineering, algorithm and hyperparameter selection to find the best model for your data. The mission: Enable automated building of machine learning with the goal of accelerating, democratizing and scaling AI. This presentation covers some recent announcements of technologies related to Automated ML, and especially for Azure. The demonstrations focus on Python with Azure ML Service and Azure Databricks.
3. Domain specific pretrained models
To simplify solution development
Azure
Databricks
Machine
Learning VMs
Popular frameworks
To build advanced deep learning solutions
TensorFlowPyTorch ONNX
Azure Machine
Learning
LanguageSpeech
…
SearchVision
Productive services
To empower data science and development teams
Powerful infrastructure
To accelerate deep learning
Scikit-Learn
Familiar Data Science tools
To simplify model development
CPU GPU FPGA
From the Intelligent Cloud to the Intelligent Edge
Azure Notebooks JupyterVisual Studio Code Command line
6. Machine Learning on Azure
Domain Specific Pretrained Models
To reduce time to market
Azure
Databricks
Machine
Learning VMs
Popular Frameworks
To build machine learning and deep learning solutions TensorFlowPyTorch ONNX
Azure Machine Learning
LanguageSpeech
…
SearchVision
Productive Services
To empower data science and development teams
Powerful Hardware
To accelerate deep learning
Scikit-Learn
PyCharm Jupyter
Familiar Data Science Tools
To simplify model development Visual Studio Code Command line
CPU GPU FPGA
From the Intelligent Cloud to the Intelligent Edge
10. How much is this car worth?
Machine Learning Problem Example
11. Model Creation Is Typically Time-Consuming
Mileage
Condition
Car brand
Year of make
Regulations
…
Parameter 1
Parameter 2
Parameter 3
Parameter 4
…
Gradient Boosted
Nearest Neighbors
SVM
Bayesian Regression
LGBM
…
Mileage Gradient Boosted Criterion
Loss
Min Samples Split
Min Samples Leaf
Others Model
Which algorithm? Which parameters?Which features?
Car brand
Year of make
12. Criterion
Loss
Min Samples Split
Min Samples Leaf
Others
N Neighbors
Weights
Metric
P
Others
Which algorithm? Which parameters?Which features?
Mileage
Condition
Car brand
Year of make
Regulations
…
Gradient Boosted
Nearest Neighbors
SVM
Bayesian Regression
LGBM
…
Nearest Neighbors
Model
Iterate
Gradient BoostedMileage
Car brand
Year of make
Car brand
Year of make
Condition
Model Creation Is Typically Time-Consuming
13. Which algorithm? Which parameters?Which features?
Iterate
Model Creation Is Typically Time-Consuming
14. Enter data
Define goals
Apply constraints
Output
Automated ML Accelerates Model Development
Input Intelligently test multiple models in parallel
Optimized model
15. Automated ML Capabilities
• Based on Microsoft Research
• Brain trained with several
million experiments
• Collaborative filtering and
Bayesian optimization
• Privacy preserving: No need
to “see” the data
16. Automated ML Capabilities
• ML Scenarios: Classification &
Regression, Forecasting
• Languages: Python SDK for
deployment and hosting for
inference – Jupyter notebooks
• Training Compute: Local
Machine, AML Compute, Data
Science Virtual Machine (DSVM),
Azure Databricks*
• Transparency: View run history,
model metrics, explainability*
• Scale: Faster model training
using multiple cores and parallel
experiments
* In Preview
17. Guardrails
Class imbalance
Train-Test split, CV, rolling CV
Missing value imputation
Detect high cardinality features
Detect leaky features
Detect overfitting
Model Interpretability / Feature Importance
19. +
To empower data science and development teams
Develop models faster with automated machine learning
Use any Python environment and ML frameworks
Manage models across the cloud and the edge.
Prepare data clean data at massive scale
Enable collaboration between data scientists and data engineers
Access machine learning optimized clusters
Azure Machine Learning
Python-based machine learning service
Azure Databricks
Apache Spark-based big-data service
20. Bring AI to everyone with an end-to-end, scalable, trusted platform
Built with your needs in mind
Support for open source frameworks
Managed compute
DevOps for machine learning
Simple deployment
Tool agnostic Python SDK
Automated machine learning
Seamlessly integrated with the Azure Portfolio
Boost your data science productivity
Increase your rate of experimentation
Deploy and manage your models everywhere
21. Fast, easy, and collaborative Apache Spark™-based analytics platform
Built with your needs in mind
Optimized Apache Spark environmnet
Collaborative workspace
Integration with Azure data services
Autoscale and autoterminate
Optimized for distributed processing
Support for multiple languages and libraries
Seamlessly integrated with the Azure Portfolio
Increase productivity
Build on a secure, trusted cloud
Scale without limits
22. Leverage your favorite deep learning frameworks
AZURE ML SERVICE
Increase your rate of experimentation
Bring AI to the edge
Deploy and manage your models everywhere
TensorFlow MS Cognitive Toolkit PyTorch Scikit-Learn ONNX Caffe2 MXNet Chainer
AZURE DATABRICKS
Accelerate processing with the fastest Apache Spark engine
Integrate natively with Azure services
Access enterprise-grade Azure security
23. What to use when?
+
Customer journey Data Prep Build and Train Manage and Deploy
Apache Spark / Big Data
Python ML developer
Azure ML service
(Pandas, NumPy etc. on AML Compute)
Azure ML service
(OSS frameworks, Hyperdrive, Pipelines,
Automated ML, Model Registry)
Azure ML service
(containerize, deploy,
inference and monitor)
Azure ML service
(containerize, deploy,
inference and monitor)
Azure Databricks
(Apache Spark Dataframes,
Datasets, Delta, Pandas, NumPy etc.)
Azure Databricks + Azure ML service
(Spark MLib and OSS frameworks +
Automated ML, Model Registry)
34. About Azure Databricks
• Azure Databricks is a managed Spark offering on Azure and customers
already use it for advanced analytics.
• It provides a collaborative Notebook based environment with CPU or
GPU based compute cluster.
35. Azure Databricks Features
• Customers who use Azure Databricks for advanced analytics can now use the
same cluster to run experiments with or without automated machine learning.
• You may keep the data within the same cluster.
• You may leverage the local worker nodes with autoscale and auto termination
capabilities.
• You may use multiple cores of your Azure Databricks cluster to perform
simultaneous training.
• You may further tune the model generated by automated machine learning.
• Every run (including the best run) is available as a pipeline, which you may tune
further if needed.
• The model trained using Azure Databricks can be registered in Azure ML SDK
workspace and then deployed to Azure managed compute (ACI or AKS) using the
Azure Machine learning SDK.
43. Try it for free
http://aka.ms/amlfree
Learn more : https://aka.ms/automatedmldocs
Notebook Samples : https://aka.ms/automatedmlsamples
Blog Post : https://aka.ms/AutomatedML
Product Feedback : AskAutomatedML@microsoft.com