This document appears to be notes from a lecture on deep learning and artificial neural networks given by Ms. Harika Pudugosula. It discusses key concepts like binary classification, linear perceptrons, feed-forward neural networks, activation functions like sigmoid and ReLU, and softmax output layers. Examples are provided to explain how neural networks can be used for problems like predicting exam performance based on sleep and study hours. Links are included at the end to additional explanatory videos on neural network topics.

1. Amrita
School
of
Engineering,
Bangalore
19ECE354 - Deep Learning
Ms. HARIKA PUDUGOSULA
Lecturer
Department of Electronics & Communication Engineering

2. Artificial Neural Networks
HARIKA 2

3. 3
HARIKA

4. 4
HARIKA

5. 5
HARIKA

6. 6
HARIKA

7. 7
HARIKA

8. 8
HARIKA

9. 9
HARIKA

10. 10
HARIKA

11. 11
HARIKA

12. 12
HARIKA

13. 13
HARIKA

14. 14
HARIKA

15. 15
HARIKA

16. 16
HARIKA

17. 17
HARIKA

18. 18
HARIKA

19. 19
HARIKA

20. 20
HARIKA

21. 21
HARIKA

22. 22
HARIKA

23. 23
HARIKA

24. 24
HARIKA

25. 25
HARIKA

26. 26
HARIKA

27. 27
HARIKA

28. 28
HARIKA

29. 29
HARIKA

30. 30
HARIKA
Binary classification
- Bipolar representation (-1/+1)
- Binary representation (0/1)

31. 31
HARIKA

32. 32
HARIKA

33. 33
HARIKA

34. 34
HARIKA

35. 35
HARIKA

36. 36
HARIKA

37. 37
HARIKA

38. 38
HARIKA

39. 39
HARIKA

40. 40
HARIKA

41. 41
HARIKA

42. HARIKA 42

43. 43
HARIKA

44. 44
HARIKA

45. HARIKA 45
Expressing Linear Perceptons
as Neurons

46. HARIKA 46
• A bias acts exactly as a weight on a connection from a unit whose
activation function is always 1
• Increasing the bias increases the net input to unit
• Fixed threshold can also be used for the activation function instead of
bias weights

47. HARIKA 47
To determine how to predict exam performance based on the number of
hours of sleep we get and the number of hours we study the previous day
• Collect a lot of data, and for each data point
x = [x1 x2]T
• Record the number of hours of sleep (x1), the number of hours spent studying (x2)
and whether performed above or below the class average
• Our goal, then, might be to learn a model h(x, θ) with parameter vector
• θ = [θ0 θ1 θ2 ]T such that

48. HARIKA 48
• Guess that the blueprint for our model
h(x, θ) is as described as a linear
classifier that divides the Cartesian
coordinate plane into two halves.
• Then, need to learn a parameter vector
θ such that our model makes the right
predictions (−1 if performance below
average, and 1 otherwise) given an
input example x
• This model is called a linear perceptron
• Then it turns out that by selecting
θ =[ −24 3 4 ]T
• The learning model makes the correct
prediction on every data point

49. HARIKA 49
• An optimal parameter vector θ positions the classifier so that we make as many
correct predictions as possible.
• Most of the time these alternatives are so close to one another that the difference
is negligible.
• If not the case , what needs to be done?
• How do we even come up with an optimal value for the parameter vector θ in the
first place?
• Solving this problem requires a technique commonly known as optimization
• An optimizer aims to maximize the performance of a learning model by iteratively
tweaking its parameters until the error is minimized.
• What happens when they is an uneven distribution of positions ?

50. HARIKA 50

51. HARIKA 51

52. HARIKA 52
Every linear perceptron can be expressed as a single neuron,
but single neurons can also express models that cannot be expressed
by any linear perceptron

53. 53
HARIKA

54. HARIKA 54
Feed-Forward Neural Networks

55. HARIKA 55
Feed-Forward Neural Networks
• A Feed Forward Neural Network is an
artificial neural network in which the
connections between nodes does not
form a cycle.
• The feed forward model is the simplest
form of neural network as information
is only processed in one direction.
• Data may pass through multiple hidden
nodes, it always moves in one direction
and never backwards.
Image Source - Quiza, Ramon & Davim, J.. (2011). Computational Methods and Optimization. 10.1007/978-1-84996-
450-0.

56. 56
HARIKA

57. HARIKA 57
Working of Feed-Forward Neural Networks
https://towardsdatascience.com/what-the-hell-is-perceptron-626217814f53

58. HARIKA 58
Linear Neuron and Their Limitations

59. 59
HARIKA

60. HARIKA 60
Sigmoid, Tanh, ReLU Neurons

61. 61
HARIKA

62. 62
HARIKA

63. 63
HARIKA

64. 64
HARIKA

65. 65
HARIKA

66. 66
HARIKA

67. 67
HARIKA

68. 68
HARIKA

69. 69
HARIKA

70. 70
HARIKA

71. 71
HARIKA

72. 72
HARIKA

73. 73
HARIKA
https://towardsdatascience.com/activation-functions-in-neural-networks-83ff7f46a6bd

74. 74
HARIKA
Softmax Output Layers

75. 75
HARIKA
https://www.analyticsvidhya.com/blog/2021/04/introduction-to-softmax-for-neural-network/

76. 76
HARIKA
• Build a neural network to recognize handwritten digits from the MNIST
dataset.
• Each label (0 through 9) is mutually exclusive, but it’s unlikely that we will be
able to recognize digits with 100% confidence.
• Using a probability distribution, the desired output vector is: where,
• This is achieved by using a special output layer called a softmax layer.
• Softmax layer depends on the outputs of all the other neurons in its layer
• Values are divided by the sum of exponential values in order to normalize and
then convert them into probabilities
• Sum of all the outputs to be equal to 1
p0 p1 p2 p3 ... p9

77. 77
HARIKA
Please go through the links below- (will be explained in the later class)
https://www.youtube.com/watch?v=8ah-
qhvaQqU&list=PLKu7faWMZGhvRokhQDmaKyw2fKA98MF_T&index=6
https://www.youtube.com/watch?v=_ETavTWv3ok&list=PLKu7faWMZGhvRokhQDmaKyw2fKA98MF_T&index=5

78. Thank you
78
HARIKA