AI-900: Microsoft Azure AI Fundamentals 2021

©Sean Xie
AI-900:
Microsoft Azure
AI Fundamentals
Sean Xie
2021

©Sean Xie
Section 1
Course & Exam Introduction

©Sean Xie
Course Introduction
• Help you prepare for the AI-900 MS Azure AI Fundamentals
• Go through all the required skills and knowledge for the exam
• Get you 100% confident to sit in the exam
Rich
Diagrams &
Pictures
Real
Samples &
Lab Results
• Illustrate core concepts & features
with multiple diagrams & samples
• Demonstrate how to create
intelligent applications with Azure
services

©Sean Xie
Exam Guide – Skills Measured
Skills/Topics Percentage
Describe Artificial Intelligence workloads and considerations
- Common AI workloads
- Guiding principles for responsible AI
15-20%
Describe fundamental principles of machine learning on Azure
- Common machine learning types
- Core machine learning concepts
- Core tasks in creating a machine learning solution
- Azure machine learning studio
30-35%
Describe features of computer vision workloads on Azure
- common types of computer vision solution
- Azure tools and services
15-20%
Describe features of Natural Language Processing (NLP) workloads on Azure
- Common NLP workload
15-20%
Describe features of conversational AI workloads on Azure
- Common use cases
15-20%
Official website: https://docs.microsoft.com/en-us/learn/certifications/exams/ai-900

©Sean Xie
Section 2
Introduction to Artificial Intelligence

©Sean Xie
“The rise of powerful AI will be
either the best or the worst thing
ever to happen to humanity.”
- Prof Stephen Hawking

©Sean Xie
What’s Artificial Intelligence (AI) – Definition from dictionaries
• “The theory and development of computer systems able to perform tasks
normally requiring human intelligence, such as visual perception, speech
recognition, decision-making, and translation between languages.”
– Oxford Dictionary
• “The study of how to produce machines that have some of the qualities that the
human mind has, such as the ability to understand language, recognize pictures,
solve problems, and learn”
– Cambridge Dictionary
• “The study of how to make computers do intelligent things that people can do,
such as think and make decisions”
– Longman Dictionary

©Sean Xie
What’s Artificial Intelligence (AI) – Definition in AI-900
• Artificial intelligence (AI) is the capability of a computer to imitate
intelligent human behavior.
Analyze Images
Comprehend Speech
Interact in Natural Ways
Detect Anomalies
Make Predictions
Vision
Hearing
Cognition
Language

©Sean Xie
Introduction to Machine Learning
• Machine Learning
A data science technique that allows computing resources to use historical
data to create a model to forecast future behaviours, outcomes, and trends.
• Purpose - Forecasts / Predictions
Register Data
Build & Train
Model
Evaluate Model Deploy Model

©Sean Xie
Machine Learning (ML) is the foundation of an AI system
Artificial
intelligence
Machine
Learning
Deep Learning
Prediction/Forecasting
Anomaly Detection
Computer Vision
Natural Language
Processing
Conversational AI
Build, Train, Deploy
Models
Neural Network

©Sean Xie
Common
AI Workloads

©Sean Xie
Prediction/Forecasting in AI
• Workloads
• Prediction of the global growth and trend of COVID-19 pandemic
• Prediction of a disease diagnosis based on symptoms and medical history
• Prediction of the future levels of carbon dioxide (CO2) in the atmosphere
• Weather prediction/forecasting
• Predict application workload and performance for resource management
• Predict sales of cars next month based on historic data
• Predict home prices next year
• Predict the travel time/fastest route using a navigation app
• Azure Services
• Machine Learning

©Sean Xie
Anomaly Detection in AI
• Workloads
• Identifying suspicious logins by looking for deviations from normal activities
• Identifying suspicious behaviors by detecting too many failed login attempts
• Monitoring machines’ temperature in the factories
• Detecting irregular heart beats using a health monitoring system in the
hospital
• Detecting fraud in credit card transactions in the Banking systems
• Azure Services
Decision
Anomaly Detector
Cognitive
Services

©Sean Xie
Computer Vision in AI – Overview
Computer Vision
Image Classification
Face Detection, Analysis & Recognition
Object Detection
Semantic Segmentation
Image Analysis
Optical Character Recognition (OCR)
Cognitive
Services
Vision
• Azure Services

©Sean Xie
Computer Vision in AI – Image Classification & Object Detection
Image Classification Object Detection
• Classify images based on contents • Classify individual objects
• Identify object’s location
• Indicate a bounding box

©Sean Xie
Computer Vision in AI – Image Analysis & Semantic Segmentation
Image Analysis Semantic Segmentation
• Extract information from images
• Return ‘tags’ with ‘confidence’
• Summarize the scene with a descriptive caption
• An advanced ML technique – label every pixel
• Locate objects and shapes
• Return a mask layer of objects

©Sean Xie
Computer Vision in AI – Face Detection & OCR
Face Detection, Analysis & Recognition Optical Character Recognition (OCR)
• Specialized form of object detection
• Locate human faces with a bounding box
• Suggest age and emotional state
• Identify or verify the identity of an individual
• Recognise text in images (photos or scanned
documents)
• Convert printed or handwritten text characters into
text data

©Sean Xie
Natural Language Processing (NLP) in AI
Natural
Language
Understanding
(NLU)
Natural
Language
Generation
(NLG)
Natural
Language
Processing
(NLP)
Language
Speech
Cognitive
Services
• Azure Services
• Read
• Interpret
• Write
• Generate

©Sean Xie
Key Phrases
Natural Language Processing (NLP) in AI – NLU
Extract Known
Entities
Determine
Sentiment
Text Interpretation Speech Recognition
• Analyze and interpret text in documents • Interpret spoken language
“Hello everyone.”
Audio Text
“To be or not to be? That is the
question. Whether TIS nobler in the
mind to suffer the slings and arrows
of outrageous fortune, or to take
arms against a sea of troubles, and by
opposing end them.”
Fantastic castle and beautiful gardens
Larnach Castle, Dunedin, New Zealand
16/12/2015
This is a unique place with more than a hundred and forty years of history. It's called New
Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We really enjoyed
our stay at Larnach Castle. Clean rooms, good service, amazing location with incredible views of
the Otago Harbour. The dinner was fantastic. We also really enjoyed exploring the gardens and
flowers. It's an unforgettable visit.

©Sean Xie
Natural Language Processing (NLP) in AI – Intent Recognition
Commands Interpretation / Intent Recognition
• Interpret commands and determine appropriate
actions
* Turn on the light please
* Please switch the lamp on
* Turn off the light please
* Please turn the lamp off
* Switch on the fan please
* Please turn the AC on
* Switch off the fan please
* Please turn the AC off
* Please close the gate
* Could you close the door
* Please open the door
* Could you open the gate

©Sean Xie
Natural Language Processing (NLP) in AI - NLG
Speech Synthesis Translation
• Synthesize speech responses • Translate spoken or written texts
Text Translation
Speech Translation
English to French:
-> Hello -> Bonjour
“Turning the light off.”
Audio
Text
“To be or not to be? That is the
question. Whether tis nobler in
the mind to suffer
The slings and arrows of
outrageous fortune, Or to take
arms against a sea of troubles
And by opposing end them.”
Text to Text
Text to Speech
Speech to Speech
Speech to Text
Hello -> Bonjour
Hello ->
->

©Sean Xie
Conversational AI
• Azure Services
• QnA Maker
• Bot Services
• AI agents participate in conversations with humans
• Conversation AI Scenarios
• Chatbots
• IBM Watson Assistant
• Home automation / Smart home
• Smart personal assistant / Virtual assistants
• Siri, Alexa, Google Assistant, Cortana, etc.
• Turing Test
Machine or Human?

©Sean Xie
Principles of
Responsible
AI

©Sean Xie
Six Ethics Principles of Microsoft Responsible AI
Fairness
Reliability &
Safety
Privacy &
Security
Inclusiveness Transparency Accountability
Responsible AI
To build a trust, reliable, safe and secure AI

©Sean Xie
Ethics Principles of Responsible AI

©Sean Xie
Artificial
Intelligence
Summary

©Sean Xie
Azure AI + ML Services in AI-900
Vision
Language
Speech
Decision
Computer Vision
Custom Vision
Face
Form Recognizer
Text Analytics
Language Understanding
QnA Maker
Speech to Text
Text to Speech
Speech Translation
Anomaly Detector
AI-900
Translator
Content Moderator
Cognitive
Services
Bot
Services
Responsible AI
Fairness
Reliability & Safety
Privacy & Security
Inclusiveness
Transparency
Accountability
ML Studio
Machine
Learning
AutoML
ML Designer
ML Workspace

©Sean Xie
Section 3
Introduction to Azure Machine Learning

©Sean Xie
Azure Machine Learning – Azure ML Service
• Azure Machine Learning service - A cloud-based platform for creating, managing,
and publishing machine learning models.
ML Studio
Azure ML
ML Workspace
Notebooks Code with Python SDK
Automated ML
ML Designer
Data & Compute Management
Pipelines
ML Types ML Core Concepts ML Core Tasks
No-code ML
No-code ML

©Sean Xie
Common
Machine
Learning
Types

©Sean Xie
Machine Learning Types – Supervised ML Models
• Supervised ML Model: training data includes known label values
Classification
predicting one of several categories
or classes
• Whether a patient is at risk of
diabetes
• whether tumor is benign or
malignant
• Categorize loan applications as
low-risk or high-risk
• Whether a loan will be repaid
• Fraud detection
• Object detection
Regression
predicting numeric values
• The number of cars that will be
sold next month
• The price of a pre-owned car
based on its maker, model,
engine size, and mileage
• The number of future car rentals
• Predicting how many minutes a
flight will be delayed based on
weather information
• The Carbon Dioxide Emissions by
Energy Consumption Use in an
area
Time-series forecasting
predicting numeric values based on
time
• Heights of ocean tides
• Weather forecasting
• Global temperature
• Monthly counts of sunspot
• Pollution levels
• Blood pressure tracking
• Heart rate monitoring
• Population
• Unemployment rates

©Sean Xie
Machine Learning Types – Unsupervised ML Models
• Unsupervised ML Model: to separate inputs into clusters based purely on
their characteristics, or features without a known label value
Clustering
group inputs into clusters based on
the similarity of their features
• Segment customers into
different groups
• Group cars of similar size or
weight
• Group penguins based their
similarities
• Image classification - group
images that have similar
characteristics
• Face grouping – divide faces into
groups Feature: Shape
The minimum number of features: 1

©Sean Xie
Machine Learning Types – Summary
Machine
Learning Types
Classification
Regression
Time-series Forecasting
Reinforcement
Learning
Supervised
Learning
Unsupervised
Learning
Clustering
Robotics
IoT

©Sean Xie
Core
Machine
Learning
Concepts

©Sean Xie
• Feature: Data values that influence the prediction of a model
• Labelling: Process of tagging training data with known values
Core ML Concepts – Features & Labels
Feature
(Input)
x
Historical Data
Label
(Output)
y
Future Outcomes
f (x) = y
Function (f)
Algorithms

©Sean Xie
Core ML Concepts – Training and Validation Datasets
• Split Data
Split Rows (one of the common techniques)
• randomly split the data into rows for
training and rows for evaluation/testing
Training
Model
Training Data Evaluation Data
Split data
• Cross-validation
• Iteratively test the trained model with data it
wasn't trained with and compare the
predicted value with the actual known value
Data
Train Model
Evaluate
Model
Training
Model
Training Data Subsets Evaluation Data
Split data
Data
Train Model
Evaluate
Model
Should NOT evaluate the
model using the same data
that was used for training

©Sean Xie
Core ML Concepts – Machine Learning Algorithms
• Microsoft Azure Machine Learning Algorithm Cheat Sheet
https://download.microsoft.com/download/3/5/b/35bb997f-a8c7-485d-8c56-19444dafd757/azure-machine-learning-algorithm-cheat-sheet-nov2019.pdf?WT.mc_id=docs-article-lazzeri
Machine
Learning
Algorithms
Multiclass
Classification
Regression
Two-Class
Classification
Clustering
Predict
Values
Predict
Categories
Group
Data
Two-Class Averaged Perceptron
Two-Class Decision Forest
Multiclass Boosted Decision
Tree
Multiclass Logistic Regression
K-Means
Image
Classification
Text Analytics
Anomaly
Detection
Recommenders
Additional Requirements
Accuracy
Training
Time
Linearity
Number of
Parameters
Number of
Features
Boosted Decision Tree
Regression
Decision Forest Regression
Fast Forest Quantile Regression
Linear Regression
Neural Network Regression
Poisson Regression
Bayesian Linear Regression
One vs. All Multiclass
Two-Class Neural Network
Two-Class Boosted Decision Tree
Two-Class Logistic Regression
Two Class Support Vector
Machine
Multiclass Decision Forest
Multiclass Neural Network
One vs. One Multiclass
Generate
Recommendations
Detect Unusual
Occurrences
Extract
Information
Classify Images

©Sean Xie
Core ML Concepts – Evaluation Metrics for Classification
True
Positives
False
Positives
False
Negatives
True
Negatives
Actual
Predicted
1 0
1
0
Confusion Matrix
Accuracy
!" + !$
!" + !$ + %" + %$
Precision
!"
!" + %"
Recall
!"
!" + %$
What’s the proportion
of correctly classified
instances?
What’s the percentage
of the class predictions
made by the model
were correct?
What’s the percentage
of a certain class
correctly identified?
The weighted average
of precision and recall
F1 Score
2 ∗ !"
2 ∗ !" + %" + %$
2*
()*+,-,./ ∗ 0*+122
()*+,-,./30*+122
=
Closer to 1
the better

©Sean Xie
Core ML Concepts – Evaluation Metrics for Classification Example
YES
NO
YES
NO
Apple Classification
Predicted Actual
TP FP
FN TN

©Sean Xie
Core ML Concepts – Evaluation Metrics for Classification –
ROC Curve & AUC
True
Positives
False
Positives
False
Negatives
True
Negatives
Actual
Predicted
1 0
1
0
Confusion Matrix
False
Positive Rate
(FPR)
!"
!" + $%
True
Positive Rate
(TPR)
$"
$" + !%
Recall
0 1
0
1
TPR
FPR
ROC Curve
(receiver operating characteristic curve)
Ideal
Bad Model Example
Random Classifier
ROC Curve Example
AUC
(area under the curve)
Ideal
Random guessing
(Y = X)
Worse than
random guessing
AUC = 1
AUC = 0.5
AUC < 0.5

©Sean Xie
Core ML Concepts – Evaluation Metrics - Predicted vs. True
• Residuals (Errors) - the difference between the predicted and actual value (label)
• Residual Histogram – a diagram shows the frequency of residual value ranges
Predicted vs. True
Residuals
Bad Model
Bad Model Example
Residual Histogram
The closer of frequently occurring residual values to 0,
more accurate the ML model.
Bad Model
Shaded Aare: Variance of predictions
around that mean predicted value

©Sean Xie
Core ML Concepts – Evaluation Metrics for Regression
Mean Absolute Error
(MAE)
Root Mean Squared Error
(RMSE)
Relative Absolute Error
(RAE)
Relative Squared Error
(RSE)
Coefficient of
Determination (R2)
The average of the absolute errors
(difference between predicted values and
true values)
!"# =
∑&'(
)
| +& − -&|
.
The square root of the mean squared
errors (difference between predicted and
true values)
The relative absolute difference between
predicted and true values
• compare models where the labels are in different
units
The relative squared difference between
predicted and true values
• compare models where the labels are in different
units
Represents the predictive power of the
model as a value between 0 and 1
RMSE =
∑/01
2 3/45/
6
)
- =
∑&'(
)
-&
.
7"# =
∑&'(
)
| +& − -&|
∑&'(
)
|- − -&|
78# =
∑&'(
)
+& − -&
9
∑&'(
) - − -&
9
79 = 1 −
∑&'(
)
+& − -&
9
∑&'(
) - − -&
9 = 1 − 78#

©Sean Xie
Core ML Concepts – Evaluation Metrics for Regression Example
Lower the value
the better
Closer to 1 the
better
Closer to 0 the
better

©Sean Xie
Machine
Learning
Solution
Core Tasks

©Sean Xie
ML Solution Core Tasks Overview
Core Tasks
Data Ingestion & Preparation
Feature Engineering & Selection
Model Training & Evaluation
Model Deployment & Management
Step 1
Step 2
Step 3
Step 4
ML Pipelines

©Sean Xie
ML Solution Core Tasks – Data Ingestion
• Create Datasets in Azure Machine Learning Designer (Recommended)
• Import Data module – import data from Azure cloud storages or public URLs
• Enter Data Manually module – create a small dataset by manually entering values
Datasets
Azure cloud storages
• Azure Blob Container
• Azure File Share
• Azure Data Lake
• Azure Data Lake Gen2
• Azure SQL Database
• Azure Database for PostgreSQL
• Databricks File System
• Azure Database for MySQL
Datastore
Local File
Azure Open
Datasets
Public URL

©Sean Xie
ML Solution Core Tasks – Data Preparation Common Tasks
Data
Transformation
Select Columns in
Dataset
Add Columns
Join Data
Remove Duplicate
Rows
Normalize Data
Clean Missing Data
Split Data
Exclude unwanted features (columns) from the dataset
Combine all columns from two or more datasets
Merge two datasets using a database-style join operation based
on the key column, or Composite keys using multiple columns.
Examples:
• Inner Join – combined rows when the values of key column match
• Left Outer Join – keep left table
Remove potential duplicates from a dataset
ID A B
01 XXX XXX
02 XXX XXX
03 XXX XXX
ID C
01 XXX
02 XXX
04 XXX
ID A B C
01 XXX XXX XXX
02 XXX XXX XXX
03 XXX XXX
ID A B
01 XXX XXX
02 XXX XXX
03 XXX XXX
ID C
01 XXX
02 XXX
04 XXX
ID A B C
01 XXX XXX XXX
02 XXX XXX XXX
Missing value/ Replacement value
Key Column

©Sean Xie
ML Solution Core Tasks – Data Preparation Common Tasks Cont.
Separate data into training and validation datasets
Examples:
• Split Rows - [0-1, default 0.5] the fraction of rows in the first output
dataset
For example, if the value is 0.7, then the dataset is divided 70/30
Data
Transformation
Select Columns in
Dataset
Add Columns
Join Data
Remove Duplicate
Rows
Normalize Data
Clean Missing Data
Split Data
Change the values of numeric columns to be on a common scale
Examples:
• MinMax - Rescaling to the [0,1]
Ignore, replace, or infer missing values
Examples:
• Minimum missing value ratio - [0-1, default 0] the minimum portion
of missing values to be operated
• Maximum missing value ratio - [0-1, default 1] the maximum portion
of missing values to be operated
• Custom substitution value
• Replace with mean / median / mode
• Remove entire row / column
Significantly larger values
may lead to higher bias

©Sean Xie
ML Solution Core Tasks – Feature Engineering & Selection
• Feature Engineering
• The process of creating new features from
original data to help machine learning algorithms
to learn better
• Automation – Featurization
• Automatically pre-process the features before training
• Feature Type: DateTime
• Engineer Features:
• Year
• Month
• Day
Features (Columns)
Observation /
Record (Rows)
• Day of week
• Day of year
• Quarter
• Week of the year
• Hour
• Minute
• Second
• Feature Selection
• The purpose is to narrow the features by selecting a subset of relevant, useful
features of original data to reduce noise and improve training performance
• The process of applying statistical tests to the input datasets to determine which
columns are more predictive of the output

©Sean Xie
Purpose
Azure ML
Module
Input /
Output
ML Solution Core Tasks – Model Training & Evaluation
Evaluate Model
Measures the accuracy of a
trained model
• Evaluate Model module – for a
regression or classification or
clustering model
• Output: evaluation metrics for
comparison of the performance
of multiple training models
Score Model
Generate predictions on new
actual data
• Score Model module – for a
regression or classification model
• Assign Data to Clusters
module – for a clustering model
• Input: dataset for validation
(Split Data)
Train Model
Train a ML model
• Train Model module – for a
regression or classification model
• Train Clustering Model
module – for a clustering model
• Input: dataset for training and
algorithm
• K-means clustering
Number of centroids parameter –
the number of clusters for the
model

©Sean Xie
ML Solution Core Tasks – Model Deployment & Management
Azure Kubernetes
Service (AKS)
Azure Container
Instance (ACI)
Development
and Testing
Production
(Real-time
Inference)
Inference Clusters
Compute Target
Trained
Model
New Data
Predict
Outcomes

©Sean Xie
ML Solution Core Tasks – Building Azure ML Pipelines
• Azure Machine Learning Pipelines
• Orchestrate data preparation, model training, deployment, and tasks management
• Help automate machine learning workflows
Training
Pipeline
Inference
Pipeline

©Sean Xie
ML Solution Core Tasks – Creating Pipelines – Regression Model
Trained Model Asset Reused
Inference Pipeline
Training Pipeline
Algorithm
Selection
Data
Preparation
Training
Scoring
Evaluation

©Sean Xie
ML Solution Core Tasks – Creating Pipelines – Classification Model
Trained Model Asset Reused
Inference Pipeline
Training Pipeline
Algorithm
Selection
Data
Preparation
Training
Scoring
Evaluation

©Sean Xie
Azure
Machine
Learning
Workspace

©Sean Xie
Azure Machine Learning – Workspace
• Azure Machine Learning Workspace
A centralized place to manage data, compute resources, code, models, and other
artefacts related to your machine learning workloads.
• Workspace Management
Task Azure Portal Azure ML Studio
Python SDK /
R SDK
Azure ML CLI
Visual Studio
Code
Create a workspace X X X X
Manage workspace access X X
Create and manage
compute resources
X X X X
Create a Notebook VM X
No Code Coding
Business/Data Experts Programmer

©Sean Xie
Azure Machine Learning – Workspace Architecture
Azure Portal
https://portal.azure.com
Azure ML Studio
https://ml.azure.com
Create
Resources Custom Vision Resource
Associated Azure Resources
Storage
Account
Container
Registry
Key Vault
Application
Insights
Custom Vision Resource
Azure ML
Workspace
Environments Experiments Pipelines Datasets Models Endpoints
Compute
Instances
Compute
Clusters
• Python packages
• Environment variables
• Software settings
• Groupings of many
runs (execution of a
training script)
• Workflows of
machine learning
tasks
• Reference to the
data source location
and its metadata
• Trained files to
recognize certain
types of patterns
• Instantiation of ML
models
• Stores artefacts & logs
• Default datastore
• Registers docker
containers
• Lazy-loaded - ACR is only
created when deployment
images are created
• Stores secrets • Stores monitoring
information about
the models
• Fully managed VMs
• Development
workstations
• Scalable clusters of VMs
• Compute targets for on-
demand processing of
experiment code
Inference
Clusters
• Deploy models to
Azure Kubernetes
Service for real-
time inferencing
Attached
Compute
• Links to existing Azure
compute resources
(VMs or Azure
Databricks clusters)
Dependencies
Managed
Resources
Assets
Manage Assets
and Resources
Azure Blob Storage
Azure File Share • Stores Jupyter
notebooks
Linked Services
Datastore
Compute Targets
Create
Workspace

©Sean Xie
Azure Machine Learning – Compute Resource
Compute
• Fully managed VMs
• Development workstations
• Run Jupyter Notebooks
• Scalable clusters of VMs
• Compute targets for on-demand
processing of experiment code
• Run Training Pipelines
• Run Inference Pipelines
• Deploy models to Azure
Kubernetes Service for real-
time inferencing (Production)
• Links to existing Azure
compute resources (VMs or
Azure Databricks clusters)

©Sean Xie
Demo
Creating an Azure ML
Workspace
in
Azure Portal

©Sean Xie
No-code
Machine Learning
with
Azure ML Studio
#1 AutoML

©Sean Xie
Azure Machine Learning – Automated ML (AutoML)
• AutoML - the process of automating the time-consuming, iterative tasks of
machine learning model development
A B C D
Trained Model
Feature Selection
Parameter Tuning
62%
75%
56%
84%
91%
86%
78%
93%
83%
59%
43%
86%
69%
75%
82%
79%
81%
70%
95%
84%
Algorithm Selection
Model Optimization Best Model
• Classification
• Regression
• Time-series Forecasting

©Sean Xie
Azure Machine Learning – AutoML Model Training Steps
Create Dataset Create Experiment Train Model Evaluate Model Deploy Model
Automated
Training
Model Selection
• Classification
• Regression
• Time series
forecasting
Upload dataset
Feature Selection
Settings
• Algorithm selection
• Parameter setting
• Featurization
Azure Kubernetes
Service (AKS)
Azure Container
Instance (ACI)
Production
(Real-time Inference)
Development &
Testing

©Sean Xie
Hands-on Lab
Training a Regression Model
with
Azure Automated ML

©Sean Xie
No-code
Machine Learning
with
Azure ML Studio
#2 Designer

©Sean Xie
Azure Machine Learning – ML Designer
• Azure ML designer provides a drag-and-drop visual canvas to build, test, and
deploy ML models.
• Datasets
• Modules
• Pipelines
• Supported open-source frameworks
• MLflow
• Kubeflow
• ONNX (Open Neural Network
Exchange)
• PyTorch
• TensorFlow
• Supported programming languages
• Python and R

©Sean Xie
Azure Machine Learning – ML Designer Model Training Steps
Create Dataset Create Training Pipeline Evaluate Model
Upload dataset
Feature Selection
Create Inference Pipeline Deploy Model
Azure Kubernetes
Service (AKS)
Azure Container
Instance (ACI)
Production
(Real-time Inference)
Development &
Testing
Feature Engineering &
Selection
Model Selection
Algorithm selection
Run the training Pipeline
Run the Inference Pipeline

©Sean Xie
Hands-on Lab
Training a Classification Model
with
Azure ML Designer

©Sean Xie
Azure
Machine
Learning
Summary

©Sean Xie
Azure Machine Learning Summary
ML Studio
ML Workspace
Automated ML
ML Designer
Data & Compute Management
Pipelines
ML Types
ML Core Concepts
ML Core Tasks
Regression
Classification
Time-series Forecasting
Reinforcement Learning
Supervised Learning
Unsupervised Learning
Clustering
Machine
Learning
Azure ML
Features & Labels
Datasets
Algorithms
Evaluations Metrics
Classification Model
Regression Model
Confusion Matrix
Accuracy
Precision
F1 Score
ROC Cure & AUC
MAE
RMSE
RAE
RSE
R2
Select Columns in Dataset
Add Columns
Join Data
Remove Duplicate Rows
Normalize Data
Clean Missing Data
Split Data
Train Model
Score Model
Evaluate Model
Training Pipeline
Inference Pipeline

©Sean Xie
Section 4
Computer Vision in Microsoft Azure

©Sean Xie
Overview of Computer Vision Services in Azure
• Azure Cognitive Services
Services
Pre-trained
Models
Custom
Models
Description
Computer Vision X Analyze content in images and videos
Custom Vision X Customize image recognition to fit your business needs
Face X X Detect and identify people and emotions in images
Form Recognizer X X Extract text, key/value pairs, and tables from documents

©Sean Xie
Computer
Vision
Service

©Sean Xie
Azure Cognitive Computer Vision Service - Overview
Image Description
Image Tags
Object Detection
Color Scheme
Image Type
Image Category
Brand Detection
Face Detection
Domain-Specific Content
Detection
Adult Content Detection
Thumbnails Generation
Optical Character
Recognition

©Sean Xie
Image Tags
• Tags based on thousands of
recognizable objects, living beings,
scenery, and actions
• Tag’s confidence
Image Description
• A caption or a list of captions ordered
from highest to lowest confidence
• based on a collection of content tags
Computer Vision Service – Image Analysis
Object Detection
• Location of the object – bounding
box’s coordinates

©Sean Xie
Computer Vision Service – Color Scheme
Image Color Scheme
• Black & White: (Boolean value)
• 1 – True
• 0 – False
• Dominant background color
• Dominant foreground color
• Dominant color of the image
• Set of dominant color (12): black,
blue, brown, gray, green, orange,
pink, purple, red, teal, white, and
yellow
• Accent color: Hexadecimal code for
HTML coding (combination of
dominant colors and saturation)

©Sean Xie
Computer Vision Service – Image Type
Image Type
• Likelihood of clip art:
• 0 – Non-clip-art
• 1 – Ambiguous
• 2 – Normal-clip-art
• 3 – Good-clip-art
• Line Drawing: (Boolean value)
• 1 – True
• 0 – False

©Sean Xie
Computer Vision Service – Image Category

©Sean Xie
Computer Vision Service – Image Category & Taxonomy
Category Taxonomy
• 86 categories
• parent/child hereditary
hierarchy
Image Category
• Categories
• Confidences
Source: https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-categorizing-images

©Sean Xie
Computer Vision Service – Brands Detection
Brands Detection
• Brand name
• Confidence score
• Location - coordinates of a
bounding box around the logo
• Logo image and brand name
can be recognized as two
separate logos

©Sean Xie
Computer Vision Service – Face Detection
Face Detection
• Age
• Gender
• Location - coordinates
of a bounding box

©Sean Xie
Computer Vision Service – Domain-Specific Content
Domain-specific Content
Celebrities
Landmarks
• Name
• Coordinates of a bounding box
around the logo
Two ways to use:
• Scoped analysis (standard)
• Enhanced categorization analysis
• Celebrities: people_ category
• Landmarks: outdoor_ or building_
categories

©Sean Xie
Computer Vision Service – Optical Character Recognition (OCR) – OCR API
OCR API
• Read a small amount of text
• Have false positive with large
amount of text
• Multiple languages (27)
• Detects Synchronously (Quick)
• Returns bounding box coordinates
• Regions that contain text
• Lines of text in each region
• Words in each line of text

©Sean Xie
Computer Vision Service – Optical Character Recognition (OCR) – Read API
Read API
• Optimized for larger documents
• Printed and handwritten text
• Supported Language: Dutch,
English, French, German,
Italian, Portuguese and Spanish
• Detects asynchronously (wait
until the analysis to complete)
• coordinates
• Pages - One for each page
of text (page size &
orientation)
• Lines of text on a page
• Words in each line of text

©Sean Xie
Computer Vision Service – Adult Content Detection
Adult Content Detection - Basic
• Categories:
• Adult – explicitly sexual in nature (e.g.,
nudity and sexual acts)
• Racy – sexually suggestive in nature (less
sexually explicit content)
• Gory – blood/gore
• Outputs
• Three Boolean value
Azure Content Moderator - Advanced
• Analyze text, image, and videos
• Moderation Outputs
• Classification Label
• Classification Score
• Operation-specific insights
• Review APIs
• Review Tools – web-based
• integrate with human reviewers
A bleeding finger protected
by a band-aid

©Sean Xie
Computer Vision Service – Thumbnails Generation
Area of Interest
• Coordinates of area of interest /
region of interest (ROI)
• Bounding box
(with Smart Cropping Enabled)
• Area of interest / region of interest
(ROI) identification
• Cropped image based on ROI
• Small version of image
• 50 x 50 pixels
Smart
Cropping
Enabled
Smart
Cropping
Disabled
Use the
center area
of image to
generate
thumbnails
Use the area
/ region of
interest of
image to
generate
thumbnails
Center area Area of Interest

©Sean Xie
Computer Vision Service – Image Criteria
• General Image Criteria:
• Supported format: JPEG, PNG, GIF, BMP
• File size: < 4 MB
• Dimensions: > 50 x 50 pixels
• OCR API:
• Dimensions: 50 x 50 to 4200 x 4200 pixels
• Read API:
• Supported format: JPEG, PNG, BMP, PDF, TIFF
• PDF and TIFF documents:
• For the free tier, only the first 2 pages are processed.
• For the paid tier, up to 2,000 pages are processed.
• Image file size: < 50 MB (4 MB for the free tier)
• Dimensions: 50 x 50 to 10000 x 10000 pixels
• PDF File Dimensions: < 17 x 17 inches (A3)

©Sean Xie
Computer Vision Service – Create Resource
Azure Portal
Computer Vision Resource
• Key
• Endpoint (HTTP address)
Model
Cognitive-Services Resource
• Key
• Endpoint (HTTP address)
Models
Option 1
Option 2
Resource Group
Options Purpose Resource Key & Endpoint
Cognitive-Services
Resource
• Use other cognitive services together
• Simplify administration and development
For all Cognitive services models
Computer Vision
Resource
• Track utilization and costs of Computer
Vision service separately
For Computer Vision model only

©Sean Xie
Hands-on Lab
Analyzing Image
with
Azure Computer Vision Service

©Sean Xie
Hands-on Lab
Analyzing Text
with
Azure Computer Vision Service

©Sean Xie
Custom
Vision
Service

©Sean Xie
Custom Vision Service – Introduction
Pre-trained
Models
Training
Model
Predication
Model
Computer Vision
Custom Vision
Organic
Recyclable
Gardening
Hazardous
Others
Waste Classification
1
2
Image Analysis
Image
Classification

©Sean Xie
Custom Vision Service – Resource & Project Creation
Azure Portal
• Key
• Endpoint (URL)
Training
Model
Predication
Model
• Key
• Endpoint (URL)
• Key
• Endpoint (URL)
Models
(Training &
Predication)
Option 1
Option 2
Custom Vision Portal
https://customvision.ai
Project
Creation
Resource Group
Image Classification Object Detection
Option 1
Option 2
Option 3 Both Training & Predication
Resource
Creation

©Sean Xie
Custom Vision Service – Image Classification Steps
Upload & Tag Images Train Model Evaluate Model Publish Model Consume Model
Tag: kangaroo
Tag: giant panda
Tag: puppy
Measure Metric:
• Precision
• Recall
• Average Precision (AP): recall & precision • Project ID
• Model name
• Predication Key
• Predication Endpoint
Quick Test
Predication
Model
Training
Model

©Sean Xie
Custom Vision Service – Object Detection Steps
Upload & Tag Images Train Model Evaluate Model Publish Model Consume Model
Measure Metric:
• Precision
• Recall
• Mean Average Precision (mAP): recall &
precision
• Project ID
• Model name
• Predication Key
Predication
Model
Training
Model
Tag: kangaroo
Tag: bird
Quick Test

©Sean Xie
Use Cases
Image Classification Use Case Description
Product Identification Identification of items/products in the supermarkets
Animals/Flowers Identification Identification of known animals/flowers in the wild
Disaster Investigation Evaluation of key infrastructure (bridges, railways, etc.) in aerial surveillance images
Medical Imaging Diagnosis Evaluation of X-ray or Magnetic resonance imaging (MRI) images
• Detection of diabetic retinopathy stages
Object Detection Use Case Description
Personal Protective Equipment
(PPE) Detection
Detection of safety helmet, glasses, goggles, gloves, face masks, face shield, high
visibility and protective workwear, etc.
Building Safety Evaluation Detection of fire extinguishers, safety signs, etc.
Self-driving Cars Lane Keeping Assist (LKA) - road markings detection
Medical Imaging Diagnosis Evaluation of X-ray or Magnetic resonance imaging (MRI) images
• Unknown object detection

©Sean Xie
Hands-on Lab
Classifying Images
with
Azure Custom Vision Service

©Sean Xie
Hands-on Lab
Detecting Object
with
Azure Custom Vision Service

©Sean Xie
Face Service – Face Detection
Face ID
• A unique identifier string
Face Attribute
• 14 attributes
• Head Pose (3D)

©Sean Xie
Face Service – Face Detection - Attribute
# Attributes Outputs
1 Age Estimated age in years
2 Gender 'male’ or 'female'
3 Smile Smile intensity (a number between [0,1])
4 Facial Hair moustache, beard, sideburns + confidences
5 Glasses 'noGlasses', 'readingGlasses', 'sunglasses', 'swimmingGoggles'
6 Head Pose 'roll', 'yaw', 'pitch’ with degrees
7 Emotion 'anger’, ‘contempt’, ‘disgust’, ‘fear’, ‘happiness’, ‘neutral’, ‘sadness’, ‘surprise’ + confidences
8 Hair ‘bald’ with confidence
Invisible (TRUE/FALSE)
Hair Color: 'unknown', 'white', 'gray', 'blond', 'brown', 'red', 'black', 'other’ + confidence
9 Makeup Eye Makeup (TRUE/FALSE); LIP Makeup (TRUE/FALSE)
10 Occlusion Forehead occluded (TRUE/FALSE)
Eye occluded (TRUE/FALSE)
Mouth occluded (TRUE/FALSE)
11 Accessories 'headWear', 'glasses', 'mask’ + confidence
12 Blur Blur Level: 'Low', 'Medium', 'High’ + a number indicating blurring level (0 to 1)
13 Exposure Exposure Level: ‘UnderExposure', 'GoodExposure', 'OverExposure’ + a number indicating exposure level (0 to 1)
- [0, 0.25) is under exposure; [0.25, 0.75) is good exposure; [0.75, 1] is over exposure
14 Noise Noise Level: 'Low', 'Medium', 'High’ + a number indicating noise level (0 to 1) - [0, 0.3) is low noise level. [0.3,
0.7) is medium noise level. [0.7, 1] is high noise level

©Sean Xie
Face Service – Face Detection - Landmarks
Face Landmarks
• 27 landmark points
• Eyebrow
• Eye
• Nose
• Mouth
• Lip
• Locations of the points
Nose Root Left Nose Root Right
Eyebrow Left Inner
Eyebrow Left Outer
Eye Left Top
Eye Left Outer
Eye Left Bottom
Eye Left Inner
Nose Tip
Nose Left Alar Top
Nose Left Out Tip
Upper Lip Top
Upper Lip Bottom
Mouth Left
Eyebrow Right Inner
Eyebrow Right Outer
Eye Right Top
Eye Right Outer
Eye Right Bottom
Eye Right Inner
Nose Right Alar Top
Nose Right Out Tip
Under Lip Top
Under Lip Bottom
Mouth Right

©Sean Xie
Face Service – Face Recognition
Verification Identification
Similarity Grouping
• Analyze the likelihood that two faces belong to the
same person
?
Do two faces belong
to the same person?
• Search and identify faces (one to many)
Person 1
Person 2
Known
Faces DB
?
Who is this
person
• Find similar faces
?
Does this person look
like other people?
• Divide faces into groups
Person 1
Person 2
Do all of these
faces belong
together?
?
Similar faces

©Sean Xie
Face Service – Face Recognition – Verification
Image Input Image Analysis
Face Service
Analysis Output
Result of
finding
similarity
Do two faces belong
to the same person?
?

©Sean Xie
• Sean
Person Group
Face Service – Face Recognition – Identification
Image Input Training Model
Face Service
Analysis Output
Prediction
Face Service
Model
Label: Sean
Label: Sean
Label: Sean
• Sean
Person Group Who is this
person
?
• Sean
Person Group

©Sean Xie
Face Service – Face Detection - Image Criteria and Performance
• Image Criteria
• Image type: JPEG, PNG, GIF, BMP
• File size: 6 MB or less
• Face size: 36 x 36 to 1920 x 1080 pixels
• Maximum detectable face size: 4096 x 4096 pixels
• Limitation - faces might not be detected
• Face size between 1920 x 1080 pixels to 4096 x 4096 pixels
• Extreme face angles (head pose) – Best results: Frontal and near-frontal faces
• Face occlusion (objects such as sunglasses that block part of the face)
• Performance Improvement
• Smoothing – turn smoothing feature off as it creates a blur between frames
• Shutter Speed – recommend shutter speeds: 1/60 second or faster to create clearer video frames
• Shutter Angle – a lower shutter angle will create a clearer video frame

©Sean Xie
Hands-on Lab
Analyzing Face
with
Azure Face Service

©Sean Xie
Form
Recognizer
Service

©Sean Xie
Form Recognizer Service – Introduction
• Features
• Identifying key/value pairs
• Extracting text
• Processing tables of data
• Identifying specific types of fields
• Receipts
• Invoices
• Business cards
• Layout extraction
• Identity Document (ID)
• Models
• Pre-built Model
• Custom Model
• Tailored to identify specific forms
• Known key/value pairs
• Known table data

©Sean Xie
Form Recognizer Service – Prebuilt Model – Receipt
Receipt Type
• Restaurants, gas stations, retail, grocery
store, etc.
key information
• Time and date of the transaction, merchant
information, amounts of taxes, line items,
transaction totals, etc.
• 15 receipt fields
• Confidence value
• Bounding box
• OCR raw text
• All text on the receipt
Supported Locales
• Australia (EN-AU), Canada (EN-CA), Great
Britain (EN-GB), India (EN-IN), and United
States (EN-US)
Customer scenarios
• Business expense reporting
• Auditing and accounting
• Consumer behavior

©Sean Xie
Form Recognizer Service – Prebuilt Model – Invoice
Extract text, key/value pairs, and
table data
key information
• customer, vendor, invoice ID, invoice
due date, total, invoice amount due,
tax amount, ship to, bill to, etc.
• 26 invoice fields
• OCR raw text
• Bounding box
• All text on the invoice

©Sean Xie
Form Recognizer Service – Prebuilt Model – Business Card
No longer used
Details no longer available
For Demo / Training purpose
key information
• Personal contact info, company
name, department, job title,
email, mobile, phone numbers,
address, website, etc.
• 13 fields
• OCR raw text
• Bounding box
• All text on the invoice
Business card sample

©Sean Xie
Form Recognizer Service – Prebuilt Model – Layout
Tables Structures Selection Marks

©Sean Xie
Form Recognizer Service – Image Criteria & Sample Tool
• Image Criteria
• Image type: JPEG, PNG, PDF, and TIFF
• File size: < 50 MB
• Face size: 50 x 50 to 10000 x 10000 pixels
• PDF and TIFF documents:
• For the free tier, only the first 2 pages are processed.
• For the paid tier, up to 2,000 pages are processed.
• PDF File Dimensions: < 17 x 17 inches (A3)
• Text: English
• Online Tools
• Invoice/Receipt/Business card/ID: https://fott-preview.azurewebsites.net/prebuilts-analyze
• Layout: https://fott-preview.azurewebsites.net/layout-analyze

©Sean Xie
Computer Vision Summary
Computer Vision
Custom Vision
Face
Form Recognizer
Cognitive
Services
Vision
Image Description
Image Tags
Object Detection
Color Scheme
Image Type
Image Category
Brand Detection
Face Detection
Domain-Specific Content Detection
Adult Content Detection
Optical Character Recognition
Image Classification
Object Detection
Face Detection
Face Recognition
Face Attribute
Face Landmarks
Verification
Identification
Similarity
Grouping
Receipts
Invoices
Business Cards
Layout
ID

©Sean Xie
Overview of NLP Services in Azure
• Azure Cognitive Services
Services
Pre-trained
Models
Custom
Models
Description
Text Analytics X Detect sentiment, key phrases, and named entities
Speech – Speech-to-Text
(Speech Recognition)
X X Transcribe audible speech into readable, searchable text
Speech – Text-to-Speech
(Speech Synthesis)
X X Convert text to lifelike speech for more natural interfaces
Speech – Speech-Translation X Integrate real-time speech translation into your apps
Translator X X Detect and translate more than 90 supported languages
Language Understanding X X
Build natural language understanding into apps, bots,
and IoT devices

©Sean Xie
Text Analytics Service – Language Detection
Review #1
16/12/2015
This is a unique place with more than a hundred and forty years of
history. It's called New Zealand's only Castle. It's a must-see place in
Dunedin. We stayed a night. We really enjoyed our stay at Larnach
Castle. Clean rooms, good service, amazing location with incredible
views of the Otago Harbour. The dinner was fantastic. We also really
enjoyed exploring the gardens and flowers. It's an unforgettable visit.
Review #2
16/12/2015
Language Detection
• Language name
• IETF BCP-47 language tag
• ISO 639-1 language code - a
two-letter lowercase
abbreviation
Examples:
• en – English
• es – Spanish
• de – German
• fr – French
• hi – Hindi
• ja – Japanese
• ko – Korean
• ru - Russian
• zh - Chinese
• Confidence score (0-1)
Language Support:
https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=language-detection

©Sean Xie
Text Analytics Service – Language Detection
Review #3
C'est incroyable - Fantastic castle and beautiful gardens
16/12/2015
This is a unique place with more than a hundred and forty years
of history. It's called New Zealand's only Castle. C'est un endroit
incontournable. We stayed a night. We really enjoyed our stay
at Larnach Castle. Clean rooms, good service, amazing location
with incredible views of the Otago Harbour. The dinner was
fantastic. We also really enjoyed exploring the gardens and
flowers. It's an unforgettable visit.
Mixed-language content
• Returns predominant
language
Ambiguous content
• Limited text or only
punctuation
• countryHint parameter
• Default countryHint –
‘US’ United States
• Language can’t be
detected
• Name/code: unknown
• Score: NaN (Not a
number)
French words
commonly used in
both English & French
aviation country_hint = 'fr'
country_hint = ’us'
illusion country_hint = 'fr'
country_hint = ’us'
Mixture of English & French
Language: unknown with confidence: NaN
Emoticons /
Kaomoji faces
:-) :) :-3 :-> 8-) :-}
(>_<) (^_^;) ^_^ (?_?)
*<|:-)

©Sean Xie
Text Analytics Service – Sentiment Analysis
Sentiment Analysis
• Label
Positive: ≥ one positive sentence + reset neutral
Negative: ≥ one negative sentence + reset neutral
Neutral: All neutral sentences
Mixed: ≥ one positive + ≥ one negative sentence
(Document Level)
• Indeterminate Sentiment
• No sufficient context or insufficient phrasing
• Wrong language code
Review #1
16/12/2015
This is a unique place with more than a hundred and forty years of history. It's called
New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a night. We
really enjoyed our stay at Larnach Castle. Clean rooms, good service, amazing
location with incredible views of the Otago Harbour. The dinner was fantastic. We
also really enjoyed exploring the gardens and flowers. It's an unforgettable visit.
Review #2
The Worst Hotel Ever
The XXX Hotel
17/12/2015
The room was dirty and the service was terrible. Our experience at this hotel was
really bad. We were shocked by the staff's unprofessional services. I won't
recommend it to anyone. Don't make the same mistake like we did.
Language Support:
https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=sentiment-analysis
0 0.5 1
Positive
Negative Neutral/
Indeterminate

©Sean Xie
Text Analytics Service – Sentiment Analysis
Text Sample:
“To be, or not to be, that is the question. The hotel is amazing, but the location is too far away from the town.
We still really enjoyed our stay at this hotel.”
Sentence #1
Sentence #2
Sentence #3
Document/Whole Text

©Sean Xie
Text Analytics Service – Key Phrase Extraction
Key Phrase Extraction
• List of main/key points
Review #1
16/12/2015
This is a unique place with more than a hundred and
forty years of history. It's called New Zealand's only
Castle. It's a must-see place in Dunedin. We stayed a
night. We really enjoyed our stay at Larnach Castle.
Clean rooms, good service, amazing location with
incredible views of the Otago Harbour. The dinner was
fantastic. We also really enjoyed exploring the
gardens and flowers. It's an unforgettable visit.
Review #1
16/12/2015
This is a unique place with more than a hundred and
forty years of history. It's called New Zealand's only
Castle. It's a must-see place in Dunedin. We stayed a
night. We really enjoyed our stay at Larnach Castle.
Clean rooms, good service, amazing location with
incredible views of the Otago Harbour. The dinner was
fantastic. We also really enjoyed exploring the
gardens and flowers. It's an unforgettable visit.
Language Support:
https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=key-phrase-extraction

©Sean Xie
Review #1
16/12/2015
This is a unique place with more than a hundred and forty years of history. It's
called New Zealand's only Castle. It's a must-see place in Dunedin. We stayed a
night. We really enjoyed our stay at Larnach Castle. Clean rooms, good service,
amazing location with incredible views of the Otago Harbour. The dinner was
fantastic. We also really enjoyed exploring the gardens and flowers. It's an
unforgettable visit.
Text Analytics Service – Entity Recognition
Review #1
16/12/2015
Entity Recognition
• Named Entity Recognition Category & Subcategories
• Person: Names
• PersonType: Job types or roles
• Location
• Geopolitical Entity (GPE) / Structural / Geographical
• Organization
• Medical / Stock exchange / Sports
• Event
• Cultural / Natural / Sports
• Product
• Computing products
• Skill
• Address
• Phone number (US and EU phone numbers only)
• Email
• URL
• IP
• DateTime
• Date / Time / DateRange / TimeRange / Duration / Set
• Quantity
• Number / Percentage / Ordinal numbers / Age /
Currency / Dimensions / Temperature
• Entity Linking - Wikipedia
• Personally Identifiable Information (PII)
• Health (PHI) (Preview)
Language Support:
https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=named-entity-recognition
https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/language-support?tabs=entity-linking

©Sean Xie
Speech Recognition vs. Speech Synthesis
• Detect and interpret spoken input • Generate spoken output
• Recorded voice -- audio file
• Live audio -- microphone
Input
Process &
Output
Acoustic
Model
audio signal
phonemes
Language
Model
words
• Text to be spoken
• Voice to be used to vocalize the speech
• Generating captions for recorded or live videos
• Creating transcripts of phone calls or meetings
• Automating note dictation
• Determining user’s input for further processing
Use Cases
Feature
Speech Recognition Speech Synthesis
• Generating spoken responses to user’s input
• Creating telephone voice menus
• Reading email or text messages
• Broadcasting announcements in public areas
• Creating voices for extinct languages
phonetic sounds
Language
Model
Acoustic
Model
individual
words prosodic units (phrases,
clauses, sentences)
phonetic transcription
Voice
(pitch, timbre)
audio signal

©Sean Xie
Speech Recognition – Speech-to-Text Service Overview
Audio files
Microphone
Custom Model
Universal
Language Model
Azure Blob
Storage
Real-time
Transcription
Batch
Transcription
(asynchronous)
Automatic Detection
Specify Language
Source
Language
Gstreamer Handling compressed audio:
MP3, OPUS/OGG, etc.
Default: WAV
Dictation
Mode Interpreting word
descriptions of sentence
structures, e.g. punctuation
• question mark à ?
Transcription
data
Pronunciation
Assessment
Evaluating the accuracy and
fluency of the audio input
Conversational
Scenarios
Multi-speaker
Multi-device
Advanced
Scenario
Language Support:
https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support#speech-to-text

©Sean Xie
Speech Synthesis – Text-to-Speech Service Overview
Text data
Real-time
Synthesis
Asynchronous
Synthesis
(Long Audio API)
Books / Lectures
(> 10 minutes)
Speech Synthesis Markup
Language (SSML)
Adjust speaking styles
(pronunciation, volume,
pitch, speaking rate, etc.)
Custom Standard
Voice
Statistical Parametric
Synthesis
Concatenation
Synthesis
Neural Voices
Deep Neural
Networks
Human-like natural
prosody
Custom Neural
Voice
Text Analyzer
Neural Acoustic
Model
Neural Vocoder
Providing phoneme
sequence
Predicting acoustic
features
Converting the
acoustic features
Audio files
Speaker
Audio files
Language Support:
https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support#text-to-speech
Stream
Standard Voices
More than 75
standard voices
Over 45 languages
and locales
Being
sunset
Viseme en-US English
(United States)

©Sean Xie
Speech Service Samples
To be or not to be? That is the question. Whether TIS nobler in the
mind to suffer the slings and arrows of outrageous fortune, or to
take arms against a sea of troubles, and by opposing end them.
Speech-to-Text Input Output
To be, or not to be, that is the question:
Whether tis nobler in the mind to suffer
The slings and arrows of outrageous fortune,
Or to take arms against a sea of troubles
And by opposing end them.
Text-to-Speech Output
Input
Standard Voice

©Sean Xie
Literal Translation vs. Semantic Translation
• Literal Translation
• word-for-word translation
• Semantic Translation
• using semantic context
Chinese
light
open
English
Open the light
Close the light
Turn on the light
Turn off the light
Results of
semantic translation
light
close
Results of
literal translation

©Sean Xie
Speech Service
Translation Services Overview
Speech Translation
Text
Translation
Speech-to-Text
Text-to-Speech
Translator Service
Speech
Translation
Translation
Text to Text
Text to Speech
Speech to Speech
Speech to Text
Use Cases:
• Document translation
• Email translation
• Webpages translation
• Customer service
translation at kiosks
Use Cases:
• Real-time closed
captioning for recorded
or live videos
• Simultaneous two-way
translation of a spoken
conversation
• Creating a transcript of
a telephone call or
meeting
Azure Services

©Sean Xie
Translation Services – Translator Service Overview
• Text-to-text translation
• Supports more than 90 languages and dialects
• Neural Machine Translation (NMT) replaces the legacy Statistical Machine
Translation (SMT)
• Supporting Custom Translator to improve translations, e.g. industry terminology
• Multiple target languages translation
• specify one from language
• multiple to languages
• Optional Configurations
• Profanity Filtering - omitting profane text translation (default is disabled)
• Selective Translation - content can be tagged so that it isn't translated
• For example, brand names
Language Support:
https://www.microsoft.com/en-us/translator/business/languages/#list

©Sean Xie
Translation Services – Speech Translation Service Overview
• Speech-to-text translation
• Speech-to-speech translation
• Translating speech to multiple target languages
• Supports more than 60 languages and dialects in real-time
• Speech recognition occurs before speech translation
Speech-to-Text Text-to-Speech
Text-to-text
translation
Speech to Speech
Translation
Speech to Text
Translation
Speech-to-Text
Text-to-text
translation
Language Support:
https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support#speech-translation

©Sean Xie
Translation Services – Translator Service Samples
Option 1: Specify source
• ‘from’: 'en-US’
Option 1: Specify one target each time, translate twice
• ‘to’: ‘zh-CN’
• ‘to’: ‘fr’
Source Language Target Language
Option 2: Language Detection
Option 2: Specify multiple targets with a comma between
the language codes, translate simultaneously
• ‘to’: ‘zh-CN’, ‘fr’

©Sean Xie
Please open the window.
Language Understanding Service – Language Understanding
• Utterances – what sentences a user might say
• Entities – an item to which an utterance refers
• Intents – purpose, goal, or task expressed in an utterance
Please open the window.
Utterance
Intent
Entity
What does the
user wants?
What are they
talking about?

©Sean Xie
Language Understanding Service – Resource & App Creation
Azure Portal
• Key
• Endpoint (URL)
Prediction
Model
Training
Model
• Key
• Endpoint (URL)
• Key
• Endpoint (URL)
Prediction
Model
Language
Understanding Portal
https://www.luis.ai/
Resource Group
Entities Creation Intents Creation
Authoring
Prediction
Option 1
Option 2
Application
Creation
Resource
Creation
Types
• Machine learned - learned from context
• List - a fixed, closed set of related words along with their
synonyms, e.g. door and gate, light and lamp
• Regex - extract an entity based on a regular expression
pattern, e.g. [0-9]{4}-[0-9]{3}-[0-9]{3} for 1234-567-890
• Pattern.any - variable-length placeholders used only in a
pattern’s template utterance to mark where entities begin
and end, e.g. Have you passed the {AI-900} exam[?]
• Intent
• NONE intent (handle utterances
that do not map any of the
utterances have been entered)
• Related Utterances (Examples)
• Labelling - Entities

©Sean Xie
Language Understanding Service – LUIS Model Building Workflow
Create Entities & Intents Train & Test Model Publish Model Consume Model
• Project ID
• Predication Key
• Predication Location
Predication
Model
• Device (Type: List)
Normalized value
• door
• light
• fan
Synonyms
• gate
• lamp
• AC
• None
• close
• open
• switch_off
• switch_on
Training
Model
Creating Intents
Creating Entities
Test Model
* Turn on the light please
* Please switch the lamp on
* Turn off the light please
* Please turn the lamp off
* Switch on the fan please
* Please turn the AC on
* Switch off the fan please
* Please turn the AC off
* Please close the gate
* Could you close the door
* Please open the door
* Could you open the gate

©Sean Xie
NLP Summary
Language
Speech
Text Analytics
Speech to Text
Text to Speech
Speech Translation
Translator
Cognitive
Services
Useful Reference:
Language Support: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support
Language Detection
Sentiment Analysis
Key Phrase Extraction
Entity Recognition
Real-time Transcription
Batch Transcription
Real-time Synthesis
Asynchronous Synthesis
Speech-to-Text Translation
Speech-to-Speech Translation
Text-to-Text Translation
Utterances
Entities
Intents

©Sean Xie
Overview of Conversational AI Services in Azure
Services
Pre-trained
Models
Custom
Models
Description
QnA Maker X X
Create a knowledge base of conversational question and
answer pairs
Azure Bot Services N/A N/A Intelligent, serverless bot services that scale on demand
Telephone
Voice Menus
Virtual Assistants
Webchat Bots
Knowledge
Base
Step one: Build a KB Step two: Build a Bot
Participate in a
conversation
Use Cases
• Webchat Bots answering common questions on
a website to provide customer services
• Telephone voice menus to reduce the load on
call center
• Virtual assistants to provide home automation

©Sean Xie
Conversational AI – QnA Maker Service
• Knowledge Base: Question and Answers Pairs
• Data Source Locations & Type
• Existing FAQ document or web page
• Entered and edited manually
• Question-answer pairs from semi-structured content, including SharePoint documents, etc.
• Pre-defined chit-chat (small talk) data source
Sample Question: When is your birthday? Source Type Content Type
URL FAQs
Support pages
PDF / DOC FAQs, Product Manual, Brochures,
Paper, Flyer Policy, Support guide,
Structured QnA, etc.
Excel Structured QnA file
TXT / TSV Structured QnA file
• Azure SQL database query is not supported

©Sean Xie
Conversational AI – QnA Maker Service – Multi-turn Conversations
• Multi-turn prompts
Question 1
Answer 1
- Prompt #1
- Prompt #2
Answer 2
- Prompt #1
- Prompt #2
- Prompt #3
Question 2
Could

©Sean Xie
Conversational AI – QnA Maker Service – Resource Setup
Azure Portal
QnA Maker
• KB ID
• QnA Auth Key
• Endpoint (URL)
Production
KB
QnA Maker Portal
https://qnamaker.ai Custom Vision Resource
Resource Group
Knowledge Base Creation
KB Creation
Resource
Creation Azure Cognitive Search
• Store QnA pairs
• Initial ranking of QnA pairs
Publish
• Enable multi-turn extraction
from URLs, .pdf or .docx files
• Add URL
• Add File
• Chi-Chat data source
Test KB
Train

©Sean Xie
Conversational AI – QnA Maker Service – KB Creation Steps
Create Knowledge Base Edit Knowledge Base Train & Test KB Publish KB
Predication
Model
Training
Model
Test Model
Define question-and-answer pairs
• Existing FAQ document
• Existing web page (URL)
• Pre-defined chit-chat data source
• Manual QnA entries
Add new
Questions &
Answers
• KB ID
• KB endpoint
• KB authorization key
QnA Maker doesn’t store
customer data (Q&A pairs
and chatlogs).
All customer data is stored in
the region of Cognitive Search
service deployed
• Alternative phrasing
• Multi-turn prompts
• Metadata tags used
to filter answer
choices

©Sean Xie
Conversational AI – Azure Bot Services
• Azure Bot
• Conversationally engages with customers
• Web application with conversational interfaces
• Three ways to create a bot:
• Bot Framework SDK - coding
• Bot Framework Composer - visual canvas
• QnA Maker automatic bot creation feature

©Sean Xie
Conversational AI – Azure Bot Services – Channels
Standard channel: Integrate a bot into a mobile app, web page, or other applications
Automatically configured when you create a bot with the Bot Framework Service
• Channel - connection between communication applications and a bot
Knowledge
Base
(QnA Pairs)
Bot
(Office 365)
WeChat
Webex
Additional Adapters
Channels
Users

©Sean Xie
Conversational AI – Azure Bot Services – Resource & Bot Setup
Azure Portal
QnA service
• KB ID
• QnA Auth Key
• Endpoint (URL)
Production
KB
QnA Maker Portal
https://qnamaker.ai Custom Vision Resource
Resource Group
Knowledge Base
Creation
KB
Creation
Resource
Creation
Azure Cognitive Search
Publish
• Enable multi-turn
extraction from URLs,
.pdf or .docx files
• Add URL
• Add File
• Chi-Chat data source
Test KB
Train
Azure Bot Service
(Web App Bot)
Web App
Bot
Custom Vision
Resource
Publishing KB
Connect Channels
1
2
3
Create Bot
Config Bot
Creating Bot

©Sean Xie
Conversational AI Summary
Language
QnA Maker
Cognitive
Services
Knowledge Base
Chi-Chat
Multi-turn Prompts
Channels
Azure Bot
Services
Web App Bot

©Sean Xie
Resources
• AI-900 Exam Official Website
https://docs.microsoft.com/en-us/learn/certifications/exams/ai-900
• Azure Documentation
https://docs.microsoft.com/en-us/azure/?product=all
• Azure SDK for Python
https://docs.microsoft.com/en-
us/python/api/overview/azure/?view=azure-python

©Sean Xie
Lab Details
Lab 01 – Creating an Azure ML Workspace
• Azure portal: https://portal.azure.com/
• Please try creating a new ML workspace in a different region if you couldn’t create a compute instance with a Free Trial
subscription. That’s why I created a second ML workspace in the region of Asia East for the following demonstrations.
Lab 02-03 – Training a Regression/Classification Model
• Azure Machine Learning Studio: https://ml.azure.com/
• Link of the lab02 exercise: https://docs.microsoft.com/en-us/learn/modules/use-automated-machine-learning/
• Link of the lab03 exercise: https://docs.microsoft.com/en-us/learn/modules/create-classification-model-azure-machine-learning-designer/
Lab 04-14 – Building Computer Vision, NLP, and Conversational AI Models
• Link of the lab exercises repository: https://github.com/MicrosoftDocs/ai-fundamentals
• Command to download the repository: !git clone https://github.com/MicrosoftDocs/ai-fundamentals
• Custom Vision Portal: https://customvision.ai
• Language Understanding Portal: https://www.luis.ai/
• QnA Maker Portal: https://qnamaker.ai
Clean-up
After you have finished the learning, please remember
to delete all the resources created during the labs and
cancel the Free-trial subscription within 30 days.
Otherwise, they can cost your money.

©Sean Xie
Thank you
• sean.xht@gmail.com
• www.linkedin.com/in/haitaoxie
It always seems impossible until it's done.
– Nelson Mandela

AI-900: Microsoft Azure AI Fundamentals 2021

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AI-900: Microsoft Azure AI Fundamentals 2021

Similar to AI-900: Microsoft Azure AI Fundamentals 2021 (20)

Recently uploaded

Recently uploaded (20)

AI-900: Microsoft Azure AI Fundamentals 2021