K fold validation

TUTORIAL ON NEURAL NET CROSSVALIDATION DESIGN
% > Reference: http://www.mathworks.com/...
% matlabcentral/newsreader/view_thread/331830#911882
% Subject: NEURAL NET CROSSVALIDATION DESIGN EXAMPLE
% From: Greg Heath Date: 29 Sep, 2013 09:01:15
% "Greg Heath" <heath@alumni.brown.edu> wrote in message
% <l28q8r$364$1@newscl01ah.mathworks.com>...
% >> "Greg Heath" <heath@alumni.brown.edu> wrote in message
% <l24d3k$ipc$1@newscl01ah.mathworks.com>...
% >>> If you find or create a NN code with stratified XVAL, please post.
%
% > I have created a stratified XVAL code for regression and curve-fitting
% using FITNET. It is applied below to the SISO SIMPLE_FIT data set. It
% uses notation that I've used before in many previous posts. In earlier
% posts I've used multi-loop codes to minimize H, the No. of hidden layer
% nodes, in order to avoid over-fitting by minimizing the ratio of unknown
% weights to training equations. This tends to result in robust designs
% that generalize well to unseen data in addition to being more resistant
% to noise and measurement error.
%
% > However, in this design I've used the default value H=10 which
% generates 31 unknown weights for SISO designs. The number of
% input/target examples for this dataset is only 94. The relatively large
% default value of H causes the ratio of equations to unknowns to be
% only 94/31 ~ 3 which is reduced to ~ 2 with the default training data
% size ratio of 70%.
% Typically, a ratio of ~10 or more is desired to achieve robustness. I
% usually deal with this by using a validation set and designing Ntrials
% ~10 or more nets for every candidate value of H. Multiple trials
% mitigates the probability of obtaining a poor initial distribution of
% random weights to begin the iterative optimization design. Then one
% or more (e.g., an ensemble) of the best nets can be used in an
% operational mode on unseen data.
%
% > An alternate approach is to use stratified k-fold cross-validation.
% MATLAB has several XVAL functions. However, they are not easily
% used for NN design, especially when the data division
Subject: TUTORIAL ON NEURAL
NET CROSSVALIDATION
DESIGN
From: Greg Heath
Date: 30 Apr, 2015 12:43:35
Message: 1 of 3
Reply to this message
Add author to My Watch List
View original format
Flag as spam
Add to My Watch
List
What is a Watch
List?
neural
crossvalida...
A tag is like a
keyword or category
label associated with
each thread. Tags
make it easier for you
to find threads of
interest.
Anyone can tag a
thread. Tags are
public and visible to
everyone.
MATLAB Central - TUTORIAL ON NEURAL NET CROSSVALID... http://www.mathworks.com/matlabcentral/newsreader/view_thread/340857
1 of 6 11/7/2015 8:44 PM

% includes a validation set (to avoid overtraining an overfit net) in
% addition to the training and testing sets. Using the defaults k=10
% and H=10, the sizes of the training, validation, and test sets will be
% 76/9/9 resulting in a 76/31 ~2.5 ratio of training equations to unknown
% weights.
%
% > The purpose of this post is to give an example of how to design
% regression/curve-fitting NNs using stratified k-fold XVAL when a
% validation set is used to prevent overtraining. The only clear
% advantage over just using multiple random designs is that the
% number of times for membership in a training, validation or test
% subset is guaranteed to be approximately equal for each data point.
%
% The purpose of this new post is to use the knowledge of the 4 local
% extrema in the smooth input/output plot to deduce a minimum lower
% value of H = 4. This results in only Nw (1+1)*4+(4+1)*1 = 13 unknown
% weights to estimate with Ntrneq = 76 training equations for a ratio of
% 76/13 ~ 5.8.
%
% In addition, the minimum gradient goal is reduced by a factor of 10.
% Therefore, if you wish to compare the current H = 4 results with the
% previous results for H = 10, you should rerun that case with the current
% code.
% However, these two results would only contain 10 designs each. To
% obtain more convincing results, I recommend increasing the number of
% designs to at least 30 each.
close all, clear all, clc, plt=0;
tic
k = 10 % k-fold XVAL
[ x , t ] = simplefit_dataset;
if iscell( simplefit_dataset) ;
x = cell2mat(x);
t = cell2mat(t);
end
[ I N ] = size(x) %[ 1 94 ]
[O N ] = size(t) % [ 1 94 ]
MSE00 = mean(var(t',1)) % 8.34 Biased Reference
MSE00a = mean(var(t',0)) % 8.43 Unbiased Reference
whos
minmaxxt = minmax( [ x ; t ] )
% minmaxxt = 0 9.9763
% 0 10
plt = plt+1, figure(plt)
plot( x, t, 'LineWidth', 2 )
title('SIMPLEFIT DATASET')
% THE SMOOTH PLOT WITH 4 LOCAL EXTREMA INDICATES
% 1. Outlier modification or removal is unnecessary
2 of 6 11/7/2015 8:44 PM

% 2. At least 4 hidden nodes are necessary
rng('default') % Or substitute your lucky number
ind0 = randperm(N);
% ind0 = 1:N; % For debugging
M = floor(N/k) % 9 length(valind & tstind)
Ntrn = N-2*M % 76 length(trnind)
Ntrneq = Ntrn*O % 76 No. of training equations
H = 4 % 4 No. of hidden nodes (default is 10)
Nw = (I+1)*H+(H+1)*O % 13 No. of unknown weights
Ndof = Ntrneq-Nw % 63 No. of estimation degrees of freedom
MSEgoal = 0.01*Ndof*MSE00a/Ntrneq % 0.069859
MinGrad = MSEgoal/100 % 0.00069859
net = fitnet(H);
net.trainParam.goal = MSEgoal;
net.trainParam.min_grad = MinGrad;
net.divideFcn = 'divideind';
% net.trainParam.max_fail = 1000; % For debugging
for i = 1:k
rngstate(i) = rng;
net = configure(net,x,t);
net.trainParam.goal = MSEgoal;
net.trainParam.min_grad = MinGrad;
% net.trainParam.max_fail = 1000; % For debugging
valind = 1 + M*(i-1) : M*i;
if i==k
tstind = 1:M;
trnind = [ M+1:M*(k-1) , M*k+1:N ];
else
tstind = valind + M;
trnind = [ 1:valind(1)-1 , tstind(end)+1:N ];
end
trnInd = ind0(trnind); % Note upper & lower case "i"
valInd = ind0(valind);
tstInd = ind0(tstind);
net.divideParam.trainInd = trnInd;
net.divideParam.valInd = valInd;
net.divideParam.testInd = tstInd;
[ net tr y e ] = train( net, x, t );
stopcrit{i,1} = tr.stop;
bestepoch(i,1) = tr.best_epoch;
R2trn(i,1) = 1 - tr.best_perf/MSE00;
R2trna(i,1) = 1 - (Ntrneq/Ndof)* tr.best_perf/MSE00a;
R2val(i,1) = 1 - tr.best_vperf/MSE00;
3 of 6 11/7/2015 8:44 PM

R2tst(i,1) = 1 - tr.best_tperf/MSE00;
end
stopcrit = stopcrit
result = [ bestepoch R2trn R2trna R2val R2tst]
minresult = min(result)
meanresult = mean(result)
medresult = median(result)
stdresult = std(result)
maxresult = max(result)
Elapsedtime = toc %3.87 sec
% stopcrit = 'Performance goal met.'
% 'Validation stop.'
% 'Performance goal met.'
% 'Minimum gradient reached.'
%
% epoch R2trn R2trna R2val R2tst
% result = 10 0.986 0.984 0.985 0.990
% 11 0.921 0.906 0.923 0.865
% 11 0.994 0.993 0.993 0.995
% 11 0.928 0.914 0.900 0.867
% 8 0.994 0.993 0.989 0.990
% 25 0.951 0.941 0.999 0.939
% 7 0.951 0.941 0.941 0.947
% 9 0.992 0.991 0.993 0.989
% 19 0.993 0.992 0.994 0.996
% 10 0.992 0.991 0.994 0.982
%
% minresult = 7.0 0.921 0.906 0.900 0.865
% meanresult = 12.1 0.970 0.964 0.971 0.956
% medresult = 10.5 0.989 0.987 0.991 0.986
% stdresult = 5.6 0.030 0.035 0.036 0.051
% maxresult = 25.0 0.994 0.993 0.999 0.996
Elapsedtime = 3.87 sec
Hope this helps.
Greg
Subject: TUTORIAL ON NEURAL Reply to this message
4 of 6 11/7/2015 8:44 PM

"Greg
Delete these equations inside the loop
DELETE THESE 3 EQUATIONS INSIDE THE LOOP
> net.trainParam.goal = MSEgoal;
> net.trainParam.min_grad = MinGrad;
> % net.trainParam.max_fail = 1000; % For debugging
THEN
> [ net tr y e ] = train( net, x, t );
WHERE
y = net(x);
e = t-y;
Hope this helps.
Greg
"Greg Heath" <heath@alumni.brown.edu> wrote in message
<mhtacn$10a$1@newscl01ah.mathworks.com>...
> "Greg
> Delete these equations inside the loop
>
> DELETE THESE 3 EQUATIONS INSIDE THE LOOP
>
> > net.trainParam.goal = MSEgoal;
> > net.trainParam.min_grad = MinGrad;
> > % net.trainParam.max_fail = 1000; % For debugging
>
> THEN
NET CROSSVALIDATION
DESIGN
From: Greg Heath
Date: 30 Apr, 2015 13:23:35
Message: 2 of 3
Flag as spam
Subject: TUTORIAL ON NEURAL
NET CROSSVALIDATION
DESIGN
From: Greg Heath
Date: 1 May, 2015 16:40:39
Message: 3 of 3
Reply to this message
Flag as spam
5 of 6 11/7/2015 8:44 PM

>
> > [ net tr y e ] = train( net, x, t );
>
> WHERE
>
> y = net(x);
> e = t-y;
>
> Hope this helps.
> Greg
It would be interesting to compare the speed and accuracy of this code with
a code that uses CROSSVAL and CVPARTITION.
Feed for this Thread
6 of 6 11/7/2015 8:44 PM

K fold validation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to K fold validation

Similar to K fold validation (20)

Recently uploaded

Recently uploaded (20)

K fold validation