make simple neural network python

!1
심층신경망 훈련
http://jaejunyoo.blogspot.com/2017/01/backpropagation.html
https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b
“Yes you should understand backprop”
"The problem with Backpropagation is that it is a leaky abstraction."
심층신경망에서는 Gradient Descent 가 잘 동작하지 않는다.
WHY?

!2
심층신경망 필요성
예측정확도
단일 = 심층
학습속도
단일 < 심층

!3
[FUNCTION] : dataset_minmax
[FUNCTION] : normalize_dataset
[FUNCTION] : evaluate_algorithm
[FUNCTION] : cross_validation_split
[FUNCTION] : back_propagation
[FUNCTION] : initialize_network
[FUNCTION] : train_network
심층신경망 학습 과정
>전처리
소스참고 : https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/
3 different varieties of wheat: Kama, Rosa and Canadian.

!4
[FUNCTION] : forward_propagate
[FUNCTION] : activate
[FUNCTION] : transfer
반복 (n_hidden + 3 회)
[FUNCTION] : backward_propagate_error
[FUNCTION] : transfer_derivative
[FUNCTION] : update_weights
심층신경망 학습 과정
> 학습
(3회 : Output layer count. In this case, 3 class classification)

!5

!6
arr[-1] : last element of arr
ㄴ s(x) sigmoid

!7

!8
def backward_propagate_error(network, expected):
# print("[FUNCTION] : ",inspect.getframeinfo(inspect.currentframe()).function)
print("-------")
pprint(network)
print(expected)
# sys.exit()
for i in reversed(range(len(network))):
layer = network[i]
errors = list()
if i != len(network)-1:
for j in range(len(layer)):
error = 0.0
for neuron in network[i + 1]:
error += (neuron['weights'][j] * neuron['delta'])
errors.append(error)
else:
neuron = layer[j]
errors.append(expected[j] - neuron['output'])
neuron = layer[j]
neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])
-------
[[{'delta': -0.002105481132614655,
'output': 0.9875157273476414,
'weights': [0.9705506423302226,
0.6235470249087426,
0.6877117204824573,
0.443648677014938,
0.5175984164765586,
0.026815578055486865,
0.6666448293753904,
0.7888475556891129]}],
[{'delta': -0.14733500412100667,
'output': 0.6043751101552216,
'weights': [0.3395066691207968, 0.08846035870939582]},
{'delta': -0.14316208422738091,
'output': 0.5598784626677785,
'weights': [0.4300707928355212, -0.1840328525536184]},
{'delta': 0.10301622967407513,
'output': 0.6026972589408445,
'weights': [0.29927233284033017, 0.12118031505837243]}]]
[0, 0, 1]
-------
[[{'delta': -0.0009853373984217329,
'output': 0.9176180659068145,
'weights': [0.9700677432652737,
0.6230543562095318,
0.6873639017456279,
0.44318707751297465,
0.5171204962344873,
0.02656030703409016,
0.6662089230964726,
0.7883548869899021]}],
[{'delta': -0.1445096162009705,
'output': 0.5651938383437569,
'weights': [0.26815390974508185, 0.016205550608910574]},
{'delta': -0.13796221046736146,
'output': 0.5197693625175647,
'weights': [0.3619508665274388, -0.2530139577872991]},
{'delta': 0.09513544172088485,
'output': 0.619274457702868,
'weights': [0.34624620530409955, 0.16874803591881485]}]]
[1, 0, 0]
-------
# Transfer neuron activation
def transfer(activation):
return 1.0 / (1.0 + exp(-activation))
# Calculate the derivative of an neuron output
def transfer_derivative(output):
return output * (1.0 - output)

!9
Sigmoid
transferred derivative
= d/dx(Sigmoid)

!10

!11

!12
네스테로프 가속 경사 추가

!13
hyper parameter test
학습률 : 0.01 학습률 : 0.10 학습률 : 0.50 학습률 : 1.00
학습률이 적을 수록 그래프가 많이 튀는 현상
“학습률이 너무 낮아서 초기화된 weight 값에서 쉽게 벗어나지 못하는 모습”

!17
~33% ~70% ~85% ~85% ~75%
for n_epoch in (3, 6, 9, 12, 15, 18, 21):
for n_hidden in (3, 6, 9, 12, 15, 18, 21):

!18
for n_epoch in (5, 10, 15, 20, 25, 30, 35):
for n_hidden in (5, 10, 15, 20, 25, 30, 35):

!19
sigmoid
> layer 가 많을 때, Gradient 소실
relu
> learning rate 에 따라 예민
ELU?

!21
# Transfer neuron activation
def transfer(activation):
## sigmoid
# return 1.0 / (1.0 + exp(-activation))
## ReLu
# return max(0, activation)
## Elu
if(activation>=0):
return activation
else:
return (exp(activation)-1)
# Calculate the derivative of an neuron output
def transfer_derivative(output):
## sigmoid
# return output * (1.0 - output)
## ReLu
# if(output > 0):
# return 1
# else:
# return 0
## Elu
if(output > 0):
return 1
else:
return (exp(output)-1)

!22
for n_epoch in (5, 10, 20, 35):
for n_hidden in (5, 10, 20, 35):

!23
for n_epoch in (5, 10, 15, 20, 25, 30, 35):
for n_hidden in (5, 10, 15, 20, 25, 30, 35):
n_epoch, n_hidden 5 5
18.6 s

make simple neural network python

Recommended

Recommended

More Related Content

Similar to make simple neural network python

Similar to make simple neural network python (20)

More from 아르떼소피

More from 아르떼소피 (7)

Recently uploaded

Recently uploaded (20)

make simple neural network python