The document discusses autoencoders, an unsupervised machine learning technique. It provides an overview of different types of autoencoders including sparse, denoising, contractive, stacked and deep autoencoders. Autoencoders learn an efficient compressed representation of input data in an unsupervised manner by training the network to ignore noise or corruptions in the input data. They are commonly used for dimensionality reduction, feature learning and compression.

1. © SuperDataScienceDeep Learning A-Z

2. © SuperDataScienceDeep Learning A-Z

3. © SuperDataScienceDeep Learning A-Z
What we will learn in this section:
• Auto Encoders
• Training of an Auto Encoder
• Overcomplete Hidden Layers
• Sparse Autoencoders
• Denoising Autoencoders
• Contractive Autoencoders
• Stacked Autoencoders
• Deep Autoencoders

4. © SuperDataScienceDeep Learning A-Z

5. © SuperDataScienceDeep Learning A-Z
Used for Regression & ClassificationArtificial Neural Networks
Used for Computer VisionConvolutional Neural Networks
Used for Time Series AnalysisRecurrent Neural Networks
Used for Feature DetectionSelf-Organizing Maps
Used for Recommendation SystemsDeep Boltzmann Machines
Used for Recommendation SystemsAutoEncoders
SupervisedUnsupervised

6. © SuperDataScienceDeep Learning A-Z
Visible
Input
Nodes
Hidden
Nodes
Visible
Output
Nodes
Encoding Decoding

7. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Visible
Input
Nodes
Hidden
Nodes
Visible
Output
Nodes
Movie 1
Movie 2
Movie 3
Movie 4

8. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Visible
Input
Nodes
Hidden
Nodes
Visible
Output
Nodes
Movie 1
Movie 2
Movie 3
Movie 4
= * (+1)
= * (-1)

9. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Visible
Input
Nodes
Hidden
Nodes
Visible
Output
Nodes
0
0
0
Movie 1
Movie 2
Movie 3
Movie 4
1 2
0
0
1
1
-2
= * (+1)
= * (-1)

10. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Visible
Input
Nodes
Hidden
Nodes
Visible
Output
Nodes
0
0
0
Movie 1
Movie 2
Movie 3
Movie 4
1 2
0
0
1
1
-2
1
0
0
0
Softmax
= * (+1)
= * (-1)

11. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Visible
Input
Nodes
Hidden
Nodes
Visible
Output
Nodes
1
0
0
Movie 1
Movie 2
Movie 3
Movie 4
0 0
2
-2
0
1
-1
0
1
0
0
= * (+1)
= * (-1)

12. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Visible
Input
Nodes
Hidden
Nodes
Visible
Output
Nodes
0
1
0
Movie 1
Movie 2
Movie 3
Movie 4
0 0
-2
2
0
-1
1
0
0
1
0
= * (+1)
= * (-1)

13. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Visible
Input
Nodes
Hidden
Nodes
Visible
Output
Nodes
0
0
1
Movie 1
Movie 2
Movie 3
Movie 4
0 -2
0
0
2
-1
-1
0
0
0
1
= * (+1)
= * (-1)

14. © SuperDataScienceDeep Learning A-Z
Neural Networks Are Impressively Good At
Compression
By Malte Skarupke (2016)
Link:
https://probablydance.com/2016/04/30/neural-networks-are-impressively-
good-at-compression/
Additional Reading:

15. © SuperDataScienceDeep Learning A-Z

16. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Movie 1
Movie 2
Movie 3
Movie 4
Bias 1
Bias 2
+1
+1
z = f(Wx + b)

17. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Movie 1
Movie 2
Movie 3
Movie 4
+1
z = f(Wx + b)
+1

18. © SuperDataScienceDeep Learning A-Z

19. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6
Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6
Visible
Input
Nodes
Hidden
Nodes
Visible
Output
Nodes
Users

20. © SuperDataScienceDeep Learning A-Z
STEP 1: We start with an array where the lines (the observations) correspond to the users and the columns (the features)
correspond to the movies. Each cell (u, i) contains the rating (from 1 to 5, 0 if no rating) of the movie i by the user u.
STEP 2: The first user goes into the network. The input vector x = (r1, r2, …, rm) contains all its ratings for all the movies.
STEP 3: The input vector x is encoded into a vector z of lower dimensions by a mapping function f (e.g: sigmoid function):
z = f(Wx + b) where W is the vector of input weights and b the bias
STEP 4: z is then decoded into the output vector y of same dimensions as x, aiming to replicate the input vector x.
STEP 5: The reconstruction error d(x, y) = ||x-y|| is computed. The goal is to minimize it.
STEP 6: Back-Propagation: from right to left, the error is back-propagated. The weights are updated according to how much
they are responsible for the error. The learning rate decides by how much we update the weights.
STEP 7: Repeat Steps 1 to 6 and update the weights after each observation (Reinforcement Learning). Or:
Repeat Steps 1 to 6 but update the weights only after a batch of observations (Batch Learning).
STEP 8: When the whole training set passed through the ANN, that makes an epoch. Redo more epochs.

21. © SuperDataScienceDeep Learning A-Z
STEP 1

22. © SuperDataScienceDeep Learning A-Z
STEP 2
Visible
Input
Nodes

23. © SuperDataScienceDeep Learning A-Z
STEP 3
Hidden
Nodes
Visible
Input
Nodes
Encoding

24. © SuperDataScienceDeep Learning A-Z
STEP 4
Hidden
Nodes
Visible
Output
Nodes
Visible
Input
Nodes
Encoding Decoding

25. © SuperDataScienceDeep Learning A-Z
STEP 5 Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6
Hidden
Nodes
Visible
Output
Nodes
Visible
Input
Nodes
Encoding Decoding

26. © SuperDataScienceDeep Learning A-Z
STEP 6 Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6
Hidden
Nodes
Visible
Output
Nodes
Visible
Input
Nodes
Encoding Decoding

27. © SuperDataScienceDeep Learning A-Z
STEP 7 Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6
Hidden
Nodes
Visible
Output
Nodes
Visible
Input
Nodes
Encoding Decoding

28. © SuperDataScienceDeep Learning A-Z
STEP 8 Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6
Hidden
Nodes
Visible
Output
Nodes
Visible
Input
Nodes
Encoding Decoding
Repeat

29. © SuperDataScienceDeep Learning A-Z
Building Autoencoders in Keras
By Francois Chollet (2016)
Link:
https://blog.keras.io/building-autoencoders-in-keras.html
Additional Reading:

30. © SuperDataScienceDeep Learning A-Z

31. © SuperDataScienceDeep Learning A-Z

32. © SuperDataScienceDeep Learning A-Z

33. © SuperDataScienceDeep Learning A-Z

34. © SuperDataScienceDeep Learning A-Z

35. © SuperDataScienceDeep Learning A-Z

36. © SuperDataScienceDeep Learning A-Z

37. © SuperDataScienceDeep Learning A-Z

38. © SuperDataScienceDeep Learning A-Z
Deep Learning Tutorial - Sparse
Autoencoder
By Chris McCormick (2014)
Link:
http://mccormickml.com/2014/05/30/deep-learning-tutorial-sparse-
autoencoder/
Additional Reading:

39. © SuperDataScienceDeep Learning A-Z
Deep Learning: Sparse Autoencoders
By Eric Wilkinson (2014)
Link:
http://www.ericlwilkinson.com/blog/2014/11/19/deep-learning-sparse-
autoencoders
Additional Reading:

40. © SuperDataScienceDeep Learning A-Z
k-Sparse Autoencoders
By Alireza Makhzani et al. (2014)
Link:
https://arxiv.org/pdf/1312.5663.pdf
Additional Reading:

41. © SuperDataScienceDeep Learning A-Z

42. © SuperDataScienceDeep Learning A-Z
1
2
3
4

43. © SuperDataScienceDeep Learning A-Z
1
2
3
4
X1
X2
X3
X4
X1
0
X3
0

44. © SuperDataScienceDeep Learning A-Z
Extracting and Composing Robust
Features with Denoising Autoencoders
By Pascal Vincent et al. (2008)
Link:
http://www.cs.toronto.edu/~larocheh/publications/icml-2008-denoising-
autoencoders.pdf
Additional Reading:

45. © SuperDataScienceDeep Learning A-Z

46. © SuperDataScienceDeep Learning A-Z
1
2
3
4

47. © SuperDataScienceDeep Learning A-Z
Contractive Auto-Encoders: Explicit
Invariance During Feature Extraction
By Salah Rifai et al. (2011)
Link:
http://machinelearning.wustl.edu/mlpapers/paper_files/ICML2011Rifai_455.pdf
Additional Reading:

48. © SuperDataScienceDeep Learning A-Z

49. © SuperDataScienceDeep Learning A-Z
Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6
Encoding Decoding
Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6

50. © SuperDataScienceDeep Learning A-Z
Encoding Decoding
Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6
Movie 1
Movie 2
Movie 3
Movie 4
Movie 5
Movie 6
Encoding

51. © SuperDataScienceDeep Learning A-Z
Stacked Denoising Autoencoders: Learning
Useful Representations in a Deep Network
with a Local Denoising Criterion
By Pascal Vincent et al. (2010)
Link:
http://www.jmlr.org/papers/volume11/vincent10a/vincent10a.pdf
Additional Reading:

52. © SuperDataScienceDeep Learning A-Z

53. © SuperDataScienceDeep Learning A-Z

54. © SuperDataScienceDeep Learning A-Z

55. © SuperDataScienceDeep Learning A-Z
Reducing the Dimensionality of Data with
Neural Networks
By Geoffrey Hinton et al. (2006)
Link:
https://www.cs.toronto.edu/~hinton/science.pdf
Additional Reading: