Downloaded 22 times
![52
#include <stdio.h>
#include <stdlib.h>
#include <limits.h>
#define TRAIN_SAMPLES 4
void main()
{
//
//training set for OR pattern
//
// input: X
double X[TRAIN_SAMPLES][2] =
{
0, 0, // 클래스 0
0, 1, // 클래스 0
1, 0, // 클래스 0
1, 1 // 클래스 1
};
// target Y
double Y[TRAIN_SAMPLES] =
{
0, 0, 0, 1
};
//
// weight
//
double W[3];
//------------------------------------------------------------
// 1) Training
//------------------------------------------------------------
//
// initialize W
//
for (int i = 0; i < 3; i++)
W[i] = ((double)rand() / RAND_MAX)*0.5 - 0.25;
unsigned int epoch = 0;
unsigned int MAX_EPOCH = 500;
double Etha = 0.05;
double output;
double x1, x2;
double target;](https://image.slidesharecdn.com/random-171207081737/85/slide-52-320.jpg)
![53
while (epoch++ < MAX_EPOCH)
{
//
// 1 epoch (for all training samples)
//
// compute deltaWi for each Wi
//
double deltaW[3];
for (int i = 0; i < TRAIN_SAMPLES; i++)
{
deltaW[0] = 0.0;
deltaW[1] = 0.0;
deltaW[2] = 0.0;
x1 = X[i][0];
x2 = X[i][1];
target = Y[i];
output = 1 * W[0] + x1 * W[1] + x2 * W[2];
deltaW[0] += (target - output) * 1;
deltaW[1] += (target - output) * x1;
deltaW[2] += (target - output) * x2;
}
//
// update W
//
W[0] = W[0] + Etha * (deltaW[0] / TRAIN_SAMPLES);
W[1] = W[1] + Etha * (deltaW[1] / TRAIN_SAMPLES);
W[2] = W[2] + Etha * (deltaW[2] / TRAIN_SAMPLES);](https://image.slidesharecdn.com/random-171207081737/85/slide-53-320.jpg)
![54
//
// compute the Cost
//
double cost;
cost = 0.0;
for (int i = 0; i < TRAIN_SAMPLES; i++)
{
x1 = X[i][0];
x2 = X[i][1];
target = Y[i];
output = 1 * W[0] + x1 * W[1] + x2 * W[2];
cost += (target - output) * (target - output);
}
cost = 0.5 * cost / TRAIN_SAMPLES;
printf("%05d: cost = %10.9lf n", epoch, cost);
}
printf("training donenn");](https://image.slidesharecdn.com/random-171207081737/85/slide-54-320.jpg)
![55
//------------------------------------------------------------
// 2) Testing for the training set
//------------------------------------------------------------
for (int i = 0; i < TRAIN_SAMPLES; i++)
{
x1 = X[i][0];
x2 = X[i][1];
target = Y[i];
output = 1 * W[0] + x1 * W[1] + x2 * W[2];
printf("%2.1lf %2.1lf (%d) %2.1lf n", x1, x2, (int)target, output);
}
printf("training test donenn");
//------------------------------------------------------------
// 3) Testing for unknown data
//------------------------------------------------------------
x1 = 0.8; // <----- 이 값을 0~1사이로 바꾸어서 테스트 해 봅니다.
x2 = 0.7; // <----- 이 값을 0~1사이로 바꾸어서 테스트 해 봅니다.
double Threshold = 0.5; // <----- 이 값을 바꾸어 최종 클래스 판단처리,
// 나중에 출력을 2개 이상으로 하면 가장 큰 값 나오는 노드를 최종 클래스로 판단
output = 1 * W[0] + x1 * W[1] + x2 * W[2];
int output_class = (output > Threshold) ? 1 : 0;
printf("%2.1lf %2.1lf (%d) %2.1lfn", x1, x2, output_class, output);
}](https://image.slidesharecdn.com/random-171207081737/85/slide-55-320.jpg)
![61
Input
Layer
Hidden
Layer
Output
Layer
Multi-Layer Perceptron
[그림] http://cs231n.github.io/neural-networks-1/](https://image.slidesharecdn.com/random-171207081737/85/slide-61-320.jpg)
![[신경망기초] 소프트맥스회귀분석](https://cdn.slidesharecdn.com/ss_thumbnails/nn09-180318142813-thumbnail.jpg?width=640&height=640&fit=bounds)
![[신경망기초] 선형회귀분석](https://cdn.slidesharecdn.com/ss_thumbnails/nn07-180318142107-thumbnail.jpg?width=640&height=640&fit=bounds)
![[신경망기초] 멀티레이어퍼셉트론](https://cdn.slidesharecdn.com/ss_thumbnails/nn04-180318141650-thumbnail.jpg?width=640&height=640&fit=bounds)
![[신경망기초] 오류역전파알고리즘](https://cdn.slidesharecdn.com/ss_thumbnails/nn06-180318141901-thumbnail.jpg?width=640&height=640&fit=bounds)
![[신경망기초] 신경망의시작-퍼셉트론](https://cdn.slidesharecdn.com/ss_thumbnails/nn02-180318141341-thumbnail.jpg?width=640&height=640&fit=bounds)
![[신경망기초] 퍼셉트론구현](https://cdn.slidesharecdn.com/ss_thumbnails/1-180604135609-thumbnail.jpg?width=640&height=640&fit=bounds)
![Backtracking [ICPC Sinchon]](https://cdn.slidesharecdn.com/ss_thumbnails/backtracking-200121122810-thumbnail.jpg?width=640&height=640&fit=bounds){kind=tracking}
![[GomGuard] 뉴런부터 YOLO 까지 - 딥러닝 전반에 대한 이야기](https://cdn.slidesharecdn.com/ss_thumbnails/part1mlp2cnn-slidesharev-180308015531-thumbnail.jpg?width=640&height=640&fit=bounds)
![[컴퓨터비전과 인공지능] 7. 합성곱 신경망 2](https://cdn.slidesharecdn.com/ss_thumbnails/lec7convolutionnetworks2-210213150820-thumbnail.jpg?width=640&height=640&fit=bounds)
![[신경망기초]오류역전파알고리즘구현](https://cdn.slidesharecdn.com/ss_thumbnails/nn-190319093725-thumbnail.jpg?width=640&height=640&fit=bounds)
![[확률통계]04모수추정](https://cdn.slidesharecdn.com/ss_thumbnails/statistics4-190131081748-thumbnail.jpg?width=640&height=640&fit=bounds)
딥러닝기본-신경망기초
- 1. 딥러닝을 위한 신경망 기초 nonezerok@gmail.com ComputerVison Pattern Recognition Lab. 1
- 2.
- 3. 3 심층신경망; deep neuralnetworks
- 4. 4 • 1957 ThePerceptron (Rosenblatt, 1957) • 1969 Perceptrons (Minsky and Papert) • 1980 Neocognitron (Fukushima, 1980) • 1982 Hopfield network, • 1982 SOM (Kohonen, 1982) • 1982 Neural PCA (Oja, 1982) • 1985 Boltzmann machines (Ackley et al., 1985) • 1986 Multilayer perceptrons and backpropagation (Rumelhart et al., 1986) • 1988 RBF networks (Broomhead&Lowe, 1988) • 1989 Autoencoders (Baldi&Hornik, 1989) • 1989 Convolutional network (LeCun, 1989, 1995) • 1992 Sigmoid belief network (Neal, 1992) • 1993 Sparse coding (Field, 1993) • 2006 DBN (Hinton, 2006) • 2012 AlexNet (2012) 신경망의 역사
- 5.
- 6.
- 7. Neural Networks Multi-Layer Perceptron DBN CNN RNN RBMAE 2-Layer Perceptron ~ Regression Linear Logistic Softmax Deep Neural Networks GAN Reinforcement Learning Supervised Learning Unsupervised Learning 7 심층학습; Deep Learning Discriminative Model Generative Model
- 8. 8 Machine Learning DataMining Decision Support System Big Data Cloud ~ Web Artificial Intelligence Image Processing Computer Vision Machine Vision Neural Networks Pattern Recognition 관련 연구분야
- 9.
- 10.
- 11.
- 12.
- 13. 13 weight height {(, )}… = − …
- 14. 14 weight height Nearest Neighbor Classification {(,)}… = − …
- 15.
- 16.
- 17. 17 모델링 수상돌기 축색돌기 축색돌기 끝 신경세포 1개를멋지게 모델링하여 만든 하나의 그림 모델
- 18.
- 19.
- 20. 20 × × ×
- 21. 21 × × + × +
- 22. 22 × + × + ×
- 23.
- 24.
- 25.
- 26. 26 = 1 +1 −1 i > 0+1 ℎ .−1 =
- 27. 27 i >0+1 ℎ .−1 , ⋯ , = 모델링 수상돌기 축색돌기 축색돌기 끝 신경세포 인공신경세포
- 28. 28 i + + ⋯ + > 0+1 ℎ .−1 , ⋯ , = i > 0+1 ℎ .−1 = 신경세포 1개를 멋지게 모델링하여 만든 하나의 수식 모델
- 29. 29 1 +1 −1 Perceptron (1957) i + + ⋯ + > 0+1 ℎ .−1 , ⋯ , = i > 0+1 ℎ .−1 =
- 30. 30 + = 0 = −1 = +1 + < 0 + > 0 직선
- 31. 31 = 0 = −1 = +1 < 0 > 0 Hyper-plane 초평면 plane 평면 + = 0 = 0 = =
- 32.
- 33. 33 Learning Perceptron 학습집합을 이용해서를 어떻게 구하지 ? = + + ⋯ + {(, )}…
- 34. 34 Learning Perceptron 방법: = + + ⋯ +
- 35. 35 차이의 합을 구해보자 ≡ 1 2 − ∈ target output
- 36. 36 Cost/Loss Function ≡ 1 2 − ∈ Squared Loss ≡ 1 2 1 − ∈ Mean Squared Loss 학습집합 크기
- 37. 37 () ∆ =음수 for ∆ =양수 for ∆ 를 어떻게 결정할까 사람은 그림을 보면 바로 판단이 선다 컴퓨터는 ?
- 38.
- 39.
- 40.
- 41. 41 - 경사하강법; Gradient DescentMethod + () 학습률; learning rate
- 42. 42 = 1 2 − = 1 2 − = 1 2 2 − − = − − −, − , − ⋯ − , = − −, = − (−1)
- 43. 43 = + ∆ ∆ = − = − , = + ∆ ∆ = − = −
- 44. 44 i + + ⋯ + > 0+1 ℎ .−1 = = + + ⋯ + = () =
- 45. • 배치 • 단일 •미니-배치 45 가중치 갱신 모드 ∆ = − , ∈ ∆ = − , ∆ = 1 − , ∈ ⊂ ℎ Gradient Descent Incremental Gradient Descent ~ gradient descent, if η is small enough Stochastic Gradient Descent
- 46.
- 47.
- 48. 48 = 1 1 =+1 = −1 1 = −1 = −1 −1 = −1 = 1 −1 = −1 C 코드 예제 ∈ −1, +1
- 49. 49 = 1 1 =1 = 0 1 = 0 = 0 0 = 0 = 1 0 = 0 ∈ 0, 1
- 50. 50 = 1 1 =1 = 0 1 = 0 = 0 0 = 0 = 1 0 = 0 = + +
- 51. 51 1 = + +
- 52. 52 #include <stdio.h> #include <stdlib.h> #include<limits.h> #define TRAIN_SAMPLES 4 void main() { // //training set for OR pattern // // input: X double X[TRAIN_SAMPLES][2] = { 0, 0, // 클래스 0 0, 1, // 클래스 0 1, 0, // 클래스 0 1, 1 // 클래스 1 }; // target Y double Y[TRAIN_SAMPLES] = { 0, 0, 0, 1 }; // // weight // double W[3]; //------------------------------------------------------------ // 1) Training //------------------------------------------------------------ // // initialize W // for (int i = 0; i < 3; i++) W[i] = ((double)rand() / RAND_MAX)*0.5 - 0.25; unsigned int epoch = 0; unsigned int MAX_EPOCH = 500; double Etha = 0.05; double output; double x1, x2; double target;
- 53. 53 while (epoch++ <MAX_EPOCH) { // // 1 epoch (for all training samples) // // compute deltaWi for each Wi // double deltaW[3]; for (int i = 0; i < TRAIN_SAMPLES; i++) { deltaW[0] = 0.0; deltaW[1] = 0.0; deltaW[2] = 0.0; x1 = X[i][0]; x2 = X[i][1]; target = Y[i]; output = 1 * W[0] + x1 * W[1] + x2 * W[2]; deltaW[0] += (target - output) * 1; deltaW[1] += (target - output) * x1; deltaW[2] += (target - output) * x2; } // // update W // W[0] = W[0] + Etha * (deltaW[0] / TRAIN_SAMPLES); W[1] = W[1] + Etha * (deltaW[1] / TRAIN_SAMPLES); W[2] = W[2] + Etha * (deltaW[2] / TRAIN_SAMPLES);
- 54. 54 // // compute theCost // double cost; cost = 0.0; for (int i = 0; i < TRAIN_SAMPLES; i++) { x1 = X[i][0]; x2 = X[i][1]; target = Y[i]; output = 1 * W[0] + x1 * W[1] + x2 * W[2]; cost += (target - output) * (target - output); } cost = 0.5 * cost / TRAIN_SAMPLES; printf("%05d: cost = %10.9lf n", epoch, cost); } printf("training donenn");
- 55. 55 //------------------------------------------------------------ // 2) Testingfor the training set //------------------------------------------------------------ for (int i = 0; i < TRAIN_SAMPLES; i++) { x1 = X[i][0]; x2 = X[i][1]; target = Y[i]; output = 1 * W[0] + x1 * W[1] + x2 * W[2]; printf("%2.1lf %2.1lf (%d) %2.1lf n", x1, x2, (int)target, output); } printf("training test donenn"); //------------------------------------------------------------ // 3) Testing for unknown data //------------------------------------------------------------ x1 = 0.8; // <----- 이 값을 0~1사이로 바꾸어서 테스트 해 봅니다. x2 = 0.7; // <----- 이 값을 0~1사이로 바꾸어서 테스트 해 봅니다. double Threshold = 0.5; // <----- 이 값을 바꾸어 최종 클래스 판단처리, // 나중에 출력을 2개 이상으로 하면 가장 큰 값 나오는 노드를 최종 클래스로 판단 output = 1 * W[0] + x1 * W[1] + x2 * W[2]; int output_class = (output > Threshold) ? 1 : 0; printf("%2.1lf %2.1lf (%d) %2.1lfn", x1, x2, output_class, output); }
- 56. 56 1 = 1, 0 = 0, 1 − 1 + 1 출력을 2개로
- 57. 57 = = 0 0 01 1 0 1 1 = = 1 0 1 0 1 0 0 1
- 58. 58 1 ⋮ = 1, 0, ⋯ , 0 = 0, 1, ⋯ , 0 1 2 = 0, 0, ⋯ , 1 ⋮ ⋮ one-hot encoding
- 59. 59 = 1 1 =1 = 0 1 = 0 = 0 0 = 0 = 1 0 = 0 퍼셉트론이 다루지 못하는 문제
- 60. 60 좀 더 복잡한문제를 다루기 위해 결과를 다시 퍼셉트론의 입력으로
- 61.
- 62. 62 = × + + × + = + + + = + + + 1 = = + 층을 늘리긴 했는데… 여전히 직선(초평면Hyper-plane)~선형
- 63. 63 1 = 선형일수 밖에 없는 이유 = () = 선형함수는 가중합만 낳을 뿐
- 64. 비선형 함수를 도입 = = 1 1 + = 1 + 64
- 65. 65 1 1 활성함수; activation function = = 1 1 +
- 66. 66 시그모이드 함수 및미분특성 = 1 1 + = 1 + = () 1 − ()
- 67. 67 = 1 1+ = 1 1 + = 1 1 + 1 − 1 1 + = 1 −
- 68. 출력층에 관한 기울기 = 1 2 − ∈ 68
- 69. 69 = + ∆ ∆ = − = 1 − − , 출력층에 관한 가중치 갱신
- 70. 70 = 1 2 − = 1 2 − = 1 2 2 − − = − − = − − = − − + , + ⋯ + , = − − 1 − , = = 1 1 + = 1 − =
- 71.
- 72. 72 = 1 2 ? − Hidden노드에서는 Target 값을 모른다
- 73. 73 오류가 전달되었다 오류에 대한가중치 변화율 = 1 2 −
- 74. 74 ∆, = , = when is of the first layer 오류 역전파 알고리즘 ∆ = − = 1 − − , ← 1 − − ← 1 − , , ← , + ∆, ℎ ,
- 75. = = − 1 − = − 1 − = 1 2 − = = + = = + = = − 1 − 1 − = 1 − = ′ 75
- 76. = = = 1 − = ′ = − 1 − = − ′ In case of output layer In case of hidden layer = = 76
- 77. = + = + = − 1 − = − 1 − = + = 1 2 − + 1 2 − 0 = + = + = − 1 − = − 1 − 0 ′ ′ 77
- 78. = + = + = + = − 1 − 1 − + − 1 − 1 − = − 1 − + − 1 − 1 − ′ 78 = + = 1 2 − + 1 2 −
- 79. = = = = − 1 − = − ′ = − 1 − = − ′ = + ′ In case of output layer In case of hidden layer 79
- 80. 80 심층신경망; deep neuralnetworks hidden 층이 2개 이상 fully connected neural networks
- 81. • Weight 수가과다 → 더 많은 수의 학습 데이터 요구됨 • Vanishing Gradient Problem 발생 → Weight 갱신이 안됨 = + ∆ ∆ = − = 1 − − − = 1 1 + = 1 − 81 심층신경망의 문제점
- 82.
- 83.
- 84. error3 error2 error1 = + (, ) (, ) (, ) = 1 2 − 84 = 1 2 + − + + − + + − (̅, )
- 85.
- 86. 86 Information, Entropy, Cross-Entropy = = = = = () 베르누이 확률변수
- 87. 87 1 1 1 × 1 1 + −
- 88. 88 −(0) 1− (1) 0 Information, Entropy, Cross-Entropy
- 89. 89 성공만 나온 경우 = = 1 = 1 실패만 나온 경우 = = 1 = 0 최댓값 성공이나 실패만 나온 경우 01 + + 1 − 1 −
- 90. 90 성공과 실패가 반반나온 경우 = = 1 = 0.5 0.5
- 91. 91 성공 70%과 실패30%나온 경우 0.7 = = 1 = 0.7
- 92. 92 Ronald Aylmer Fisher(1890~1962) cross-entropy = negative maximum log likelihood estimator
- 93.
- 94. Logit function 94 입력 출력 0 ~1 +∞ −∞
- 95. Logit의 출력을 신경망출력으로 간주 95 입력 출력 +∞ −∞
- 96. 96 입력과 출력을 뒤바꾸면…
- 97. 97 신경망의 출력을 확률로간주할 수 있다! + +∞−∞
- 98.
- 99. + = 1 1 + 99
- 100. + = 1 1 + 100
- 101. Logistic Regression for2-class Classification 101 1
- 102. Logistic Regression for2-class Classification 102 Loss function: = − ∑ + 1 − 1 − Activation () = ∈ 0, 1 베르누이 시행, 확률변수 1
- 103. Softmax Regression forMulti-class Classification 103 1 ⋮ + + ⋯ + + + ⋯ + + + ⋯ +
- 104. Softmax Regression forMulti-class Classification 104 Loss function: = − ∑ ∑ () = ∑ = 0 0 1 0 0 0 0 0 0 0 1 ⋮ + + ⋯ + + + ⋯ + + + ⋯ + Softmax Function
- 105. 105 = + ∆ ∆ = − = 1 − − ,
- 106.
- 107. 107 V h2 h1 O w1 w2 w3 V h2 h1 O w1 w2 w3 V h2 h1 O w1 w2 w3 Pre-Training by Layer-WiseTraining with RBM RBM을 이용한 선행 학습, 미세조정
- 108. 108 Auto-Encoder를 이용한 선행학습 1 2
- 109.
- 110. 110 가중치 공유; weightsharing 1 2주변만 연결 공유 36 16 4
- 111. 111 1 2주변만 연결공유 36 16 4 합성곱; convolution 연산
- 112.
- 113.
- 114. 114 특징 맵; featuremap 입력영상, 가중치의 합성곱의 결과 feature map
- 115.
- 116. 116 CNN in Practicefor MNIST fully-connected http://neuralnetworksanddeeplearning.com/index.html
- 117. 117 Rectified Liner Unit(ReLU) 1 = 0 > 0 ℎ
- 118. 118https://nn.readthedocs.io/en/rtd/transfer/ 활성함수 미분 sigmoid hypertangent ReLU = + ∆ ∆ = − = − 1 − , = () 1 − () vanishing gradient problem
- 119. 119 데이터 설정, 학습,평가 http://cs231n.github.io/
- 120.
- 121. 적응적 학습률 http://cs231n.github.io/neural-networks-3/ Momentum ∆ = − 1 − , + ∆ − 1 good high low very high epoch loss 학습률
- 122.
- 123. 학습, 검증, 테스트분석 123 epoch accuracy training validation; little overfitting validation; strong overfitting
- 124. 124 정규화; regularization = 1 2 − ∈ + 1 2 = − + 1 2 regularization strength 1 2 L2-regularization L1-regularization = 0.01~0.0001
- 125. 125 = + ∆ ∆ = − = − 1 − − −, + 가중치 감쇠; weight decay 가중치는 자신의 크기에 비례하여 줄어듦 바이어스에 대해서는 정규화 사용하지 않음
- 126. 126 Dropout ~ Ensemble Dropout:A simple way to prevent neural networks from overfitting." The Journal of Machine Learning Research 15.1 (2014): 1929-1958.
- 127.
- 128. 가중치, 편향치 초기화 128 LeCun XavierGlorot & Bengio (2010) He (2015) = 0, 0.01 good for sigmoid better for PReLU good for ReLU
- 129.
- 130. • Tom Mitchell,Machine Learning, McGraw Hill, 1997. • Nielsen, Michael, Neural Networks and Deep Learning, 2015. • http://neuralnetworksanddeeplearning.com/index.html • CS231n Convolution Neural Networks for Visual Recognition • http://cs231n.github.io/ 130 참고문헌
- 131.
- 132.
- 133. 133 요슈아 벤지오 얀 르쿤리차드 서튼 마이클 조던 블라디미르 뱁닉 이안 굿펠로우앤드류 응