Introduction to Machine Learning

Presentation Introduction Machine Learning Deep Learning
Introduction to Machine Learning
Machine Learning and Deep Learning
Tommy Löfstedt
Umeå University, Umeå, Sweden
tommy.lofstedt@umu.se
October 22, 2019
Tommy Löfstedt — Introduction to Machine Learning 1/62

Introduction
Decomposition of the AI field
Deep learning
Representation learning
Machine learning
AI

Introduction
Artificial Intelligence
Fundamental idea:
Intelligent agent (brain)
Perceives inputs from environment (environment)
Maps inputs to actions (body)
Artificial intelligence (AI) is a huge field
Logic
Probability
Reasoning
Planning
Action
Philosophy
Perception
Learning

Introduction
Machine learning
“Machine learning is the subfield of computer
science that gives computers the ability to learn
without being explicitly programmed.”
— Arthur Samuel, 1959

Introduction
Machine learning
Definition
A computer program is said to learn from experience E with
respect to some class of tasks T and performance measure P,
if its performance at tasks in T, as measured by P, improves
with experience E.
— Mitchell (1997). “Machine Learning”.

Introduction
Machine learning
Traditional computer programming:
Inputs Program Computer Outputs
Machine learning:
Computer
Inputs
Outputs
Program

Introduction
Machine learning
Take-home idea: Learn from data
Use case: When manually programming something is not
feasible
Autonomous vehicles
Speech recognition
Natural language processing
Computer vision
etc.
How then? Use some data to achieve some maximal score
on some task!

Introduction
Machine learning
Example:
T: Classify a tumour as malignant or benign
P: Fraction of correctly classified tumours (accuracy)
E: Set of training data and ground truth outputs

Machine Learning
Two main types of learning
Supervised learning
Labelled/annotated data
Example input-output pairs, {(xi , yi )}, i = 1, . . . , n
Learn to predict output from input, H 3 h ≈ f : X → Y
Unsupervised learning
No labels
Only “input” data, {xi }, i = 1, . . . , n
Find underlying latent structure in the input data

Machine Learning
Two (four) main types of learning
Supervised
Regression
Classification
Unsupervised
Clustering
Dimensionality reduction
(And many more . . . )

Machine Learning
What is required?
Define an hypothesis space, H
Formulate loss function (cost, error, objective, risk, etc.)
to evaluate choices:
` : P → R
Collect data
Input: independent variable, variable, feature, covariate,
predictor, factor, etc.
Output: dependent variable, outcome, target, etc.
Minimise the loss function over the hypothesis space
using the data

Machine Learning
Example: Regression
Relationship between input and output: f : X → Y
In most cases Y = R
Linear regression has
H 3 h(x; β) =
p
X
j=1
xj βj + β0
Linear regression uses the mean squared error loss:
`(β) =
1
n
n
X
i=1
yi − h(xi ; β)
2
We thus attempt
minimise
β∈Rp,β0∈R
`(β)

Machine Learning
Example: Regression

Machine Learning
Example: Classification
Relationship between input and output: f : X → C
In most cases C is a finite set of categories
Example: Logistic regression
Two classes: C = {0, 1}
Denote: pi = P(yi = 1)
Assume: log pi
1−pi
=
Pp
j=1 xij βj + β0

Machine Learning
Straight-forward algebra gives:
P(yi = 1) = pi =
1
1 + e−
Pp
j=1 xij βj +β0
=: σ
p
X
j=1
xij βj + β0
!
−10.0 −7.5 −5.0 −2.5 0.0 2.5 5.0 7.5 10.0
logits
0.0
0.2
0.4
0.6
0.8
1.0
σ(logits)
The Logistic Sigmoid Function

Machine Learning
Family of hypotheses:
H 3 h(xi ; β) = σ
p
X
j=1
xij βj + β0
!
Logistic regression uses the binary cross-entropy loss:
`(β) = −
1
n
n
X
i=1
yi log(h(xi ; β)) + (1 − yi ) log(1 − h(xi ; β))
We again seek
minimise
β∈Rp,β0∈R
`(β)

Machine Learning

Machine Learning
Example: Clustering
Relationships amongst the inputs: f : X → C
In most cases C is a finite set of cluster indices
Input: objects with associated distance function
Output: a cluster, or likelihood to belong to each cluster
Example: K-means clustering (Lloyd’s algorithm)
Assume K clusters: C = {1, . . . , K}
The clusters have means µk for k = 1, . . . , K
Let h(x; µ1, . . . , µK ) = arg mink∈C kx − µkk2

Machine Learning
Example: Clustering
H 3 h(x; µ1, . . . , µK ) = arg min
k∈C
kx − µkk2
K-means clustering minimises the within-cluster sum of
squares:
`(µ1, . . . , µK , C1, . . . , CK ) =
K
X
k=1
X
x∈Ck
kx − µkk2
2
We seek to
minimise
µ1,...,µK ∈X
`(µ1, . . . , µK , C1, . . . , CK )

Machine Learning
Example: Clustering

Machine Learning
Example: Dimensionality Reduction
Reducing the input space: f : X → b
X, with b
X ⊂ X
Feature selection
Feature extraction
Example: Principal Component Analysis
Assumption: The data are well-approximated by a
d-dimensional linear subspace, d p

Machine Learning
Family of hypotheses: H 3 h(x; P) = PT
x
Principal component analysis maximises variance:
`(P) = −Var(h(x; P))
We seek
minimise
P
`(P)
subject to pT
i pj = 0, i 6= j
kpi k2
2 = 1 ∀i = 1, . . . , d

Machine Learning

Machine Learning
What have we learned?
We have some problem that we cannot solve manually
We have some data ({(xi , yi )})
Select a machine learning method (H)
Select a loss function (`)
Minimise the loss over the hypothesis space
Ok, easy! Are we done?

Machine Learning
The Variance-Bias Trade-Off and Model Selection
Example: Polynomial regression

Machine Learning
Problem: We have
P(|`(h) − EX [`(h)]| ε) ≤ 2De−2ε2n
for D ∝ p
What can we do?
Collect more data (increase n)
Ask less of your model (increase ε)
Reduce the input space (dim. reduction, decrease p)
Constrain/regularise the hypothesis space (decrease D)

Machine Learning
Constrain/regularise the hypothesis space
minimise
β∈P
`(β)
subject to ϕ(β) ≤ C
⇐⇒
minimise
β∈P
`(β) + λϕ(β)

Machine Learning
Example: Polynomial ridge regression
minimise
β∈Rp
1
n
ky − Xβk2
2 + λkβk2
2
where Xi = [1, xi , x2
i , . . . , xp−1
i ].

Machine Learning

Short break?
Next Up: Deep Learning

Deep Learning
History
Wang and Raj (2017)

Deep Learning
History
1940–1970: Cybernetics
Biological learning
Mimicking the brain
The perceptron (1 neuron)
1980–1995: Connectionism
Back-propagation algorithm
Up to a couple of hidden layers
2006–Now: Deep learning
Pre-training using restricted Boltzmann machines
Deep belief networks
Tens or hundreds (thousands!) of hidden layers

Deep Learning
Introduction
Traditionally, a human provided the features (6=data)
Machine learning method maximised the score on the task
Now, “the machine” discovers the mapping from data to
features, and the mapping from features to output
End-to-end system
Data
Feature
Extractor
Machine
Learning
Outputs
Data
Feature
Learning
Machine
Learning
Outputs
Deep Learning

Deep Learning
What is it?
Basic idea:
Features are learned from raw data
Features are learned from features
A hierarchy of features
Lower layers in the hierarchy contains
“simpler” features
Higher layers in the hierarchy contains
“complex” features
Lines and blobs in images combines to
form objects

Deep Learning
Neural Networks: Inspiration from Neuroscience
Original idea: Recreate the building
blocks of the brain
Your brain:
1011 neurons
1014 synapses
1017 ops./sec.
Dendrites receive information
Soma “processes” the information
Action potential if enough signal
from other neurons
Sends processed information
through axon

Deep Learning
An Artificial Neural Network
Fundamental idea:
“Artificial neurons” receive
information
Sums the inputs
Sends inputs through “activation
function”, φ
Outputs processed information
y = φ β0 +
p
X
j=1
xj βj
!

Deep Learning
An Artificial Neural Network
y1 = φ(XW0 + b0)
yi+1 = φ(yi Wi + bi ), i = 1, 2, . . . , L
ŷ = ϕ(yLWL + bL)
ŷ = ϕ(φ(· · · (φ(φ(XW0 + b0)W1 + b1) · · · )WL−1 + bL−1)WL + bL)

Deep Learning
Example: A Shallow Artificial Neural Network for Regression
One hidden layer: y1 = φ(xW1 + b1)
ŷ = ϕ(y1W2 + b2)
Rectified linear unit: Let φ(x) = max(0, x)
“Linear” output activation: Let ϕ(x) = x
H 3 h(x; W1, b1, W2, b2) = ϕ(φ(xW1 + b1)W2 + b2)
We then minimise the mean squared error loss:
`(W1, b1, W2, b2) =
1
n
n
X
i=1
yi − h(xi ; W1, b1, W2, b2)
2

Deep Learning
Example: A Shallow Artificial Neural Network for Regression
−1.0 −0.5 0.0 0.5 1.0
−0.2
0.0
0.2
0.4
0.6
Nodes: 0, MSE: 0.030
−1.0 −0.5 0.0 0.5 1.0
−0.2
0.0
0.2
0.4
0.6
−1.0 −0.5 0.0 0.5 1.0
−0.2
0.0
0.2
0.4
0.6
−1.0 −0.5 0.0 0.5 1.0
−0.2
0.0
0.2
0.4
0.6
−1.0 −0.5 0.0 0.5 1.0
−0.2
0.0
0.2
0.4
0.6
Nodes: 10, MSE: 0.006
−1.0 −0.5 0.0 0.5 1.0
−0.2
0.0
0.2
0.4
0.6
Nodes: 1000, MSE: 0.004

Deep Learning
Example: Artificial Neural Networks for Classification
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
Linear, Accuracy: 0.51
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
Nodes: 2, Accuracy: 0.66
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
Nodes: 3, Accuracy: 0.91
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
Nodes: 6-2, Accuracy: 0.91
−1.0 −0.5 0.0 0.5 1.0
−1.00
−0.75
−0.50
−0.25
0.00
0.25
0.50
0.75
1.00
Last hidden layer

Deep Learning
Example: Artificial Neural Networks for Classification
What’s really going on here?
X XW1 φ(XW1)
φ(XW1)W2 φ(φ(XW1)W2)

Deep Learning
Stochastic Gradient Descent—The power-house of deep learning
Intuition:
Compute network output
Determine how each parameter
affected the error
Update each parameter such that
the error is reduced
Repeat

Deep Learning
Stochastic Gradient Descent—The power-house of deep learning
β(k+1)
= β(k)
− α(k)
∇f (β(k)
)
Gradient, ∇f , computed by back-propagation
Expensive to compute ∇f for large data sets
∇f (β|X) ≈
1
n
n
X
i=1
∇f (β | xi ), n 1
→ E[∇f (β)], when n → ∞
Insight: E[∇f (β)] = E[∇f (β|xi )]
Mini-batches to reduce noise
Results indicate it is part of the success of deep learning!
Provides a regulariser for the solution

Deep Learning
The success
Amount of available training data
Computer hardware (GPUs)
Computer software (tensor/graph libraries)
Improved regularisation techniques
Dropout
Batch normalisation
Data augmentation
Small perturbations of the data
Scaling, translations, rotations, reflections, etc.
Cropping, noise, linear combinations of samples, etc.

Deep Learning
The success
Deeper models
Transfer learning
Better initialisation
Better training algorithms
Better models
Residual Networks
Densely Connected
Networks
U-Net
Skip connections!
Li et al. (2017)

Deep Learning
Convolutional Neural Networks
Introduced by LeCun et al. (1989)
ILSVRC2012: Krizhevsky et al. (2012)
Revolutionised image analysis
Also in speech recognition, synthesis, etc.
Very similar to fully connected layers
y1j = φ

b0j +
X
i
Xi ∗ W0ij


Deep Learning
The Convolution Operator

Deep Learning
Gabor-like filters found in mammals (e.g. in cats)
Gabor-like filters appear in the first layers of CNNs
Combined to detect complex shapes

Deep Learning
Zeiler and Fergus (2013)

Deep Learning

Deep Learning
Ren et al. (2016):
Choi et al. (2017):

Deep Learning
Nvidia (2015):
Kontschieder et al. (2017):

Deep Learning
Karpathy and Fei-Fei (2015):
Gatys et al. (2015):

Deep Learning
Convolutional Neural Networks in Medical Imaging
Roughly a 2.2 times increase of papers per year since 2011

Deep Learning
Generative adversarial networks (GANs)
Two networks play a min-max game
A generator network outputs samples
A discriminator network distinguishes generated samples
from real samples
z xfake
G(z)
pθ(z)
xreal
pdata(x)
x real?
D(x)

Deep Learning
Progressive GANs
A recent development of GANs (Karras et al., 2018)
The generator and discriminator are grown progressively
Provides fast and stable training of GANs

Deep Learning
StyleGAN
State of the art for human face synthesis
Examples: http://thispersondoesnotexist.com

Deep Learning
Image Synthesis using the StyleGAN
Random example images generated using the StyleGAN model

Deep Learning
Image Synthesis using the StyleGAN
Example images generated from the MR and CT parts of the
disentangled space (W)

Deep Learning
Reinforcement Learning
Multi-agent hide and seek by OpenAI (2019)

Deep Learning
Reinforcement Learning
Robot hand solving Rubik’s cube by OpenAI (2019)

The end
Thank you for your attention!
Questions?

This training material is part of the FogGuru project that has received funding
from the European Union’s Horizon 2020 research and innovation programme
under the Marie Skłodowska-Curie grant agreement No 765452. The information
and views set out in this material are those of the author(s) and do not
necessarily reflect the official opinion of the European Union. Neither the
European Union institutions and bodies nor any person acting on their behalf
may be held responsible for the use which may be made of the information
contained therein.

Introduction to Machine Learning

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Introduction to Machine Learning

Similar to Introduction to Machine Learning (20)

More from FogGuru MSCA Project

More from FogGuru MSCA Project (20)

Recently uploaded

Recently uploaded (20)

Introduction to Machine Learning