The document provides an introduction to machine learning and deep learning, outlining its goals, foundational disciplines, and various tasks such as supervised and unsupervised learning. It discusses the evolution of machine learning methods, including rule-based systems, classic machine learning, and deep learning, along with practical applications like spam filtering and image recognition. Additionally, it highlights prominent tools, libraries, courses, and online resources for further learning in the field.

1. Introduction to
Deep Learning
Dmytro Fishman, University of Tartu

2. What is Machine Learning?
Was born with a goal to create Artificial Intelligence
!
There were two main branches:
• Symbolic AI, based on if/then like rules (GOFAI)
!
• Inspired by Neuroscience principle - learn through
examples (later developed into ML)

3. Machine Learning
founding disciplines
Computer
Science
Statistics
Information
Theory Databases
Probability
Theory
Optimisation
Machine Learning

4. What is Machine Learning?
Machine learning is a scientific field that develops
computer programs that make data-driven decisions
without being explicitly programmed and also
automatically improve over time.

5. What is Machine Learning?
Machine learning is
• a scientific field
• develops computer programs
• make data-driven decisions
• without being explicitly programmed to
• also automatically improve over time.

6. Examples of Machine Learning
Spam Filtering

7. Examples of Machine Learning
Face Recognition

8. Examples of Machine Learning
Recommendation Systems

9. Comes from nature…
http://www.borgenmagazine.com/wp-content/uploads/2014/06/natural_disasters.jpg

10. …and industry
http://forums.vwvortex.com/showthread.php?3208922-2008-Dodge-Challenger-Factory-production-pics./page2

11. Often is very messy
http://insights.wired.com/profiles/blogs/what-s-the-big-deal-with-unstructured-data

12. Ideally a long !
list of examples
of the same
nature

13. Types of data
255,255,255,255
255,255,255,255
255, 56, 65,255
255, 98, 89,110
To be or not to be.
x = (0.5, 0.9, 0.7, -0.3, …)
x = (255, 255, 255, 255, …)
x = (0, …, 0, 2, 1, 0, 0, 1, 2, …)
to
or
not
be
the
cat
low
John
Adopted from P.Vincent http://videolectures.net/deeplearning2015_vincent_machine_learning/

14. Machine Learning tasks
Supervised Learning
• Classification
• Regression!
!
Unsupervised Learning!
• Clustering
• Dimensionality reduction
• Density Estimation
!
Reinforcement Learning - taking actions in an
environment to maximise reward

15. Machine Learning tasks
Supervised Learning
• Classification
• Regression!
!
Unsupervised Learning!
• Clustering
• Dimensionality reduction
• Density Estimation
!
Reinforcement Learning - taking actions in an
environment to maximise reward

16. Supervised Learning
Cat!
Cat
???
Dog
x1 = (87, 205, 255, 155, …); y1 = 1
x2 = (95, 195, 245, 155, …); y2 = 1
x3 = (43, 159, 255, 5, …); y3 = -1
x4 = (180, 66, 245, …); !
y4 = f(x4) = 1
Features Labels
Adopted from P.Vincent http://videolectures.net/deeplearning2015_vincent_machine_learning/

17. Evolution of ML methods
Rule-based
Classic Machine
Learning
Representation
Learning
Deep Learning
Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/
Input Input Input Input
Fixed set of
rules
Output
Hand designed
features
Learning
Automated
feature extraction
Output
Learning
Output
Low level
features
High level
features
Learning
Output

18. Mimicking the brain
Engineering perspective
• Compact!
• Energy efficient (12 watts—a fifth of
the power of a lightbulb)
• 10^11 neurons
• for each neurone there are 10-50
supporting cells (glial cells)
• 10^14 synapses (connections)
Computing power
• Fairly slow in arithmetic calculations
• Very efficient in computer vision, speech, language etc.
Adopted from L.Bottou http://videolectures.net/deeplearning2015_bottou_neural_networks/

19. Biological Neuron

20. Caution!
Here comes the slide with
some math!

21. Artificial Neurone
x0
w0
x1w1
x2w2
X
i
xiwi + b f
axon from neurone
dendrite
synapse
output axon
activation!
function
f(
X
i
xiwi + b)
Slide from CS231n: Convolutional Neural Networks for Visual Recognition http://cs231n.github.io/ by Andrej Karpathy

22. Feed Forward
x0
w0
x1w1
x2w2
X
i
xiwi + b f
axon from neurone
dendrite
synapse
output axon
activation!
function
f(
X
i
xiwi + b)
Slide from CS231n: Convolutional Neural Networks for Visual Recognition http://cs231n.github.io/ by Andrej Karpathy
class Neuron:
# ...
def forward(inputs):
""" assume inputs and weights are 1-D numpy arrays and bias is a number """
cell_body_sum = np.sum(inputs * self.weights) + self.bias
firing_rate = 1.0 / (1.0 + math.exp(-cell_body_sum)) # sigmoid activation
function
return firing_rate

23. Training Neural Netowrks
43
255
32
Output
Hidden
…
Input
Image

24. 32
143
255
Output
Hidden
…
Input
Image
Training Neural Netowrks

25. 32
143
255
Output
Hidden
…
Input
Image
Training Neural Netowrks

26. 0.6
32
143
Input
255
0.4
Output
Hidden
Image
…
Cat
Dog
Error
Training Neural Netowrks

27. Deep Neural Networks

28. GoogLeNet

29. Types of Deep Neural
Networks
• Convolutional Neural Networks (CNN)
• Convolutional Deep Belief Networks (CDBN)
!
• Recurrent Neural Networks (RNN)
• Long Short-Term Memory Networks (LSTM)
!
https://en.wikipedia.org/wiki/Deep_learning

30. Convolutional Neural
Networks

31. ImageNet Large Scale Visual Recognition
Challenge (ILSVRC)

32. Breed?
http://cs.stanford.edu/people/karpathy/ilsvrc/

33. Misclassified examples
Overall:
trained expert got 5.1% of errors
GoogLeNet 6.7% of errors
Before Deep Neural Networks the best
result was 20.7%

34. Galaxy Zoo Challenge
61578 JPG images, made by Hubble

35. Galaxy Zoo Challenge
http://benanne.github.io/2014/04/05/galaxy-zoo.html

36. Diabetic Retinopathy Detection
Challenge
0 stage
4 stage
and 80 000 more images

37. Diabetic Retinopathy Detection Challenge

38. …
…
Diabetic Retinopathy Detection Challenge

39. DeepMind

40. DeepMind

41. 0.6
32
143
Input
255
0.4
Output
Hidden
Image
…
Right
Left
Lost
one
life?
Training DeepMind

42. Recurrent Neural
Networks

43. Shakespeare?
Recurrent Neural Network
PANDARUS:
Alas, I think he shall be come approached and the day
When little srain would be attain'd into being never fed,
And who is but a chain and subjects of his death,
I should not sleep.
!
Second Senator:
They are away this miseries, produced upon my soul,
Breaking and strongly should be buried, when I perish
The earth and thoughts of many states.
!
DUKE VINCENTIO:
Well, your wit is in the care of side and that.
!
Second Lord:
They would be ruled after this chamber, and
my fair nues begun out of the fact, to be conveyed,
Whose noble souls I'll have the heart of the wars.
!
Clown:
Come, sir, I will make did behold your worship.
!
VIOLA:
I'll drink it.
VIOLA.
On your attendance, my lord; here.
!
DUKE.
Stand you awhile aloof.—Cesario,
Thou know'st no less but all; I have unclasp'd
To thee the book even of my secret soul:
Therefore, good youth, address thy gait unto her;
Be not denied access, stand at her doors,
And tell them there thy fixed foot shall grow
Till thou have audience.
!
VIOLA.
Sure, my noble lord,
If she be so abandon'd to her sorrow
As it is spoke, she never will admit me.
!
DUKE.
Be clamorous and leap all civil bounds,
Rather than make unprofited return.
!
VIOLA.
Say I do speak with her, my lord. What then?
4.4МB
Shakespeare
texts
Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy

44. Recurrent Neural Network
PANDARUS:
Alas, I think he shall be come approached and the day
When little srain would be attain'd into being never fed,
And who is but a chain and subjects of his death,
I should not sleep.
!
Second Senator:
They are away this miseries, produced upon my soul,
Breaking and strongly should be buried, when I perish
The earth and thoughts of many states.
!
DUKE VINCENTIO:
Well, your wit is in the care of side and that.
!
Second Lord:
They would be ruled after this chamber, and
my fair nues begun out of the fact, to be conveyed,
Whose noble souls I'll have the heart of the wars.
!
Clown:
Come, sir, I will make did behold your worship.
!
VIOLA:
I'll drink it.
VIOLA.
On your attendance, my lord; here.
!
DUKE.
Stand you awhile aloof.—Cesario,
Thou know'st no less but all; I have unclasp'd
To thee the book even of my secret soul:
Therefore, good youth, address thy gait unto her;
Be not denied access, stand at her doors,
And tell them there thy fixed foot shall grow
Till thou have audience.
!
VIOLA.
Sure, my noble lord,
If she be so abandon'd to her sorrow
As it is spoke, she never will admit me.
!
DUKE.
Be clamorous and leap all civil bounds,
Rather than make unprofited return.
!
VIOLA.
Say I do speak with her, my lord. What then?
Shakespeare?
Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy

45. Linux
474MB of the
source code
/*!
* Increment the size file of the new incorrect UI_FILTER group information!
* of the size generatively.!
*/!
static int indicate_policy(void)!
{!
int error;!
if (fd == MARN_EPT) {!
/*!
* The kernel blank will coeld it to userspace.!
*/!
if (ss->segment < mem_total)!
unblock_graph_and_set_blocked();!
else!
ret = 1;!
goto bail;!
}!
segaddr = in_SB(in.addr);!
selector = seg / 16;!
setup_works = true;!
for (i = 0; i < blocks; i++) {!
seq = buf[i++];!
bpf = bd->bd.next + i * search;!
if (fd) {!
current = blocked;!
}!
}!
rw->name = "Getjbbregs";!
bprm_self_clearl(&iv->version);!
regs->new = blocks[(BPF_STATS << info->historidac)] | PFMR_CLOBATHINC_SECONDS << 12;!
return segtable;!
}
Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy

46. Tools
• Python (many implementations)
• Torch (http://torch.ch/)
• Theano (http://deeplearning.net/software/theano/)
• Caffe (http://caffe.berkeleyvision.org/)
!
!
Deep Learning Libraries by programming language:
http://www.teglor.com/b/deep-learning-libraries-
language-cm569

47. Online courses
CS224d: Deep Learning for Natural Language
Processing
by Richard Socher
http://cs224d.stanford.edu/
Machine Learning
by Andrew Ng
https://www.coursera.org/learn/machine-learning
Neural Networks for Machine Learning
by Geoffrey Hinton
https://www.coursera.org/course/neuralnets
CS231n: Convolutional Neural Networks for
Visual Recognition
by Andrej Karpathy
http://cs231n.github.io/

48. Offline places
Computer Science Bachelor’s program
at CS@UCU
http://cs.ucu.edu.ua/en/
Applied Computer Science Stream
at Summer School AACIMP (Kyiv)
http://summerschool.ssa.org.ua/
The Deep Learning Summer School
at Montreal, Canada
https://sites.google.com/site/deeplearningsummerschool/
Computer Science Master’s program
at Tartu, Estonia
https://sites.google.com/site/deeplearningsummerschool/
International Conference on Machine Learning
at New York, USA
http://icml.cc/2016/

49. Popular Blogs
10 Machine Learning Terms Explained in Simple English
by Aylien
http://blog.aylien.com/post/121281850733/10-machine-learning-terms-explained-in-simple
The Unreasonable Effectiveness of
Recurrent Neural Networks
by Andrej Karpathy
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
A Neural Network in 11 lines of Python
by iamtrask
http://iamtrask.github.io/2015/07/12/basic-python-network/
Understand LSTM Networks
by Christopher Olah
https://colah.github.io/posts/2015-08-Understanding-LSTMs/
Sign up for Data Elixir mailing list: http://dataelixir.com/

50. Kaggle
Platform for data science competitions
Sponsored
Research
Public competitions

51. Computers are useless.
They can only give
answers.
Pablo Picasso