Deep Learning Fundamental Concepts

1. Natural Neural Networks,
Artificial Neural Networks,
and Deep Learning
Fundamental
Concepts
Farrokh Karimi

2. Natural Neural Networks

3. An early map of functional localization in the brain.

4. Rostral-caudal and dorsal-ventral axes

5. Section planes

6. Various slides of a 3D image

7. The divisions of the central nervous system

8. The main cortical and subcortical regions can
be imaged in the brain of a living individual
A. Schematic drawing shows B. Magnetic Resonance Image (MRI)

9. The structure of a neuron

10. A microscopic image of a neuron

11. Basic neural circuit motifs

12. Visual processing

13. Pupillary reflex and accommodation Eye movement (horizontal)

14. Visual pathways in the macaque monkey

15. The eye projects the visual scene onto the
retina’s photoreceptors
Refraction of light onto the retina
Focusing of light in the fovea
Packing of photoreceptors
in the fovea

16. Representation of
the visual field along
the visual pathway

17. Comparison of biological and machine learning
networks

18. Expectation and perceptual task play a critical
role in what is seen

19. Priming image for the Figure

20. Illusory contours and perceptual fill-in

21. Illusory contours and perceptual fill-in

22. A test for some forms of color blindness
The numerals embedded in this color pattern can be distinguished by people with trichromatic vision but not by
dichromats who are weak in red–green discrimination. If you don’t see any numbers please have your vision tested.

23. Artificial Neural Networks

25. Artificial intelligence, machine learning, and deep learning

26. Major tasks of AI
• Supervised Learning (Classification, Regression, SVM, KNN, …)
• Learn Model from both input(s) and output(s) data
• Input(s): Object Properties
• Output(s): Discrete or “Limited Set” (e.g. Labels) or Continuous
• Unsupervised Learning (Clustering/Structure Finding, …)
• Not desired output!
• Find structure of data/Clustering/Grouping/…
• Semi-Supervised Learning
• Reinforcement learning
• …

27. Classification
• Input(s) are objects: Electric Signals/Image/Speech/DNA
sequences/Email/….
• Output(s) are binary (mostly Labels): “benign”/”malignant” (Tumor),
“Primary”, “Social”, “Promotion” (Gmail)

28. Regression
• Input(s) are objects: Electric Signals/Image/Speech/DNA
sequences/Email/….
• Output(s) are Continuous: Temperature, Score, Fuel Consumption, …

29. Support Vector Machines (SVM)
• A margin based classifier/regressor

30. Clustering
• Input(s) are objects
• No Output data!

31. Single layer Perceptron

32. Representational power of perceptron

33. Representational power of perceptron

34. Some linearly separable functions

35. XOR is not linearly separable

36. Multilayer Perceptron

37. Decision regions of a multilayer neural network

38. Gradient descent

39. Gradient descent
Error

40. Learning rate issue

41. Learning rate issue

42. Complex problems space

43. Other Activation functions

44. The sigmoid activation function
• Smooth gradient
• Non-Zero at origin
• Vanishing gradient
• Computationally expensive

45. Tanh function
• Same as sigmoid
• Zero at origin

46. ReLu (Rectified Linear)
• Non-linear
• Computationally efficient
• Inactive near origin or negative

47. SoftMax (multioutput network)

48. Evaluation

49. Train – Test Split

50. Train – Validation – Test Split

51. Cross validation

52. Cross validation example

53. Confusion matrix for 2-class problems

54. Confusion matrices for multi-class problems

55. Other accuracy metrics

56. ROC curves

57. ROC curve example

58. Other accuracy metrics
Sensitivity

59. Regression Loss/Cost Function

60. Bias-Variance Dilemma

61. Typical Performance
or
Number of Epochs

62. Overfitting Underfitting problems

63. Overfitting Underfitting problems

64. Stopping criteria

65. References
• Principles of Neural Science (6th) – Eric Kandel
• Neural Networks (2nd) - A Comprehensive Foundation – Simon Haykin

66. Thanks
https://farrokhkarimi.github.io/