Successfully reported this slideshow.
Upcoming SlideShare
×

# Neural Networks

• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

### Neural Networks

2. 2. 1-parameter linear regression Assume that the data is formed by yi = wxi + noisei where… • the noise signals are independent • the noise has a normal distribution with mean 0 and unknown variance σ2 P(y|w,x) has a normal distribution with • mean wx • variance σ2 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 3 Bayesian Linear Regression P(y|w,x) = Normal (mean wx, var σ2) We have a set of datapoints (x1,y1) (x2,y2) … (xn,yn) which are EVIDENCE about w. We want to infer w from the data. P(w|x1, x2, x3,…xn, y1, y2…yn) •You can use BAYES rule to work out a posterior distribution for w given the data. •Or you could do Maximum Likelihood Estimation Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 4
3. 3. Maximum likelihood estimation of w Asks the question: “For which value of w is this data most likely to have happened?” <=> For what w is P(y1, y2…yn |x1, x2, x3,…xn, w) maximized? <=> For what w is n ∏P( y w, x ) maximized? i i i =1 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 5 For what w is n ∏ P( y w , x i ) maximized? i i =1 For what w is n 1 ∏ exp(− 2 ( yi − wxi ) 2 ) maximized? σ i =1 For what w is 2 1  y − wx i  n ∑ −i  maximized? σ 2  i =1 For what w is 2 n ∑ (y ) − wx minimized? i i i =1 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 6
4. 4. Linear Regression The maximum likelihood w is the one that E(w) w minimizes sum- Ε = ∑( yi − wxi ) 2 of-squares of residuals i (∑ x )w = ∑ yi − (2∑ xi yi )w + 2 2 2 i i We want to minimize a quadratic function of w. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 7 Linear Regression Easy to show the sum of squares is minimized when ∑x y i i w= ∑x 2 i The maximum likelihood Out(x) = wx model is We can use it for prediction Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 8
5. 5. Linear Regression Easy to show the sum of squares is minimized when ∑ xi yi p(w) w= w ∑x 2 Note: i In Bayesian stats you’d have ended up with a prob dist of w The maximum likelihood Out(x) = wx model is And predictions would have given a prob dist of expected output Often useful to know your confidence. We can use it for Max likelihood can give some kinds of prediction confidence too. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 9 Multivariate Regression What if the inputs are vectors? 3. .4 6. 2-d input 5 . example .8 x2 . 10 x1 Dataset has form x1 y1 x2 y2 x3 y3 .: : . xR yR Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 10
6. 6. Multivariate Regression Write matrix X and Y thus:  .....x1 .....   x11 x1m   y1  x12 ... .....x .....  x  y  x22 ... x2 m   =  21 x= y =  2 2    M M M     .....x R .....  xR1 xR 2 ... xRm   yR  (there are R datapoints. Each input has m components) The linear regression model assumes a vector w such that Out(x) = wTx = w1x[1] + w2x[2] + ….wmx[D] The max. likelihood w is w = (XTX) -1(XTY) Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 11 Multivariate Regression Write matrix X and Y thus:  .....x1 .....   x11 x1m   y1  x12 ... .....x .....  x ... x2 m   x22  y =  y2   =  21 x= 2    M M M     .....x R .....  xR1 xR 2 ... xRm   yR  IMPORTANT EXERCISE: (there are R datapoints. Each input hasPROVE IT !!!!! m components) The linear regression model assumes a vector w such that Out(x) = wTx = w1x[1] + w2x[2] + ….wmx[D] The max. likelihood w is w = (XTX) -1(XTY) Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 12
7. 7. Multivariate Regression (con’t) The max. likelihood w is w = (XTX)-1(XTY) R ∑x x ki kj XTX is an m x m matrix: i,j’th elt is k =1 R ∑x y XTY is an m-element vector: i’th elt ki k k =1 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 13 What about a constant term? We may expect linear data that does not go through the origin. Statisticians and Neural Net Folks all agree on a simple obvious hack. Can you guess?? Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 14
8. 8. The constant term • The trick is to create a fake input “X0” that always takes the value 1 X1 X2 Y X0 X1 X2 Y 2 4 16 1 2 4 16 3 4 17 1 3 4 17 5 5 20 1 5 5 20 Before: After: Y=w1X1+ w2X2 Y= w0X0+w1X1+ w2X2 …has to be a poor = w0+w1X1+ w2X2 In this example, You should be able model …has a fine constant to see the MLE w0 , w1 and w2 by term inspection Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 15 Regression with varying noise • Suppose you know the variance of the noise that was added to each datapoint. y=3 σ=2 σi2 xi yi ½ ½ 4 y=2 σ=1/2 1 1 1 σ=1 2 1 1/4 y=1 σ=1/2 σ=2 2 3 4 y=0 3 2 1/4 x=0 x=1 x=2 x=3 MLE the w? yi ~ N ( wxi , σ i2 ) at’s f W h at e o Assume stim e Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 16
9. 9. MLE estimation with varying noise argmax log p( y , y ,..., y | x1 , x2 ,..., xR , σ 12 , σ 2 ,..., σ R , w) = 2 2 1 2 R w Assuming i.i.d. and ( yi − wxi ) 2 R then plugging in argmin ∑ = equation for Gaussian σ i2 and simplifying. i =1 w   x ( y − wx ) Setting dLL/dw R  w such that ∑ i i 2 i = 0  = equal to zero   σi   i =1  R xi yi  Trivial algebra ∑ 2     i =1 σ i   R xi2  ∑ 2   σ  i =1 i  Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 17 This is Weighted Regression • We are asking to minimize the weighted sum of squares y=3 σ=2 ( yi − wxi ) 2 R argmin ∑ σ i2 y=2 i =1 σ=1/2 w σ=1 y=1 σ=1/2 σ=2 y=0 x=0 x=1 x=2 x=3 1 where weight for i’th datapoint is σ i2 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 18
10. 10. Weighted Multivariate Regression The max. likelihood w is w = (WXTWX)-1(WXTWY) xki xkj R ∑ (WXTWX) is an m x m matrix: i,j’th elt is σ i2 k =1 R xki yk (WXTWY) is an m-element vector: i’th elt ∑ σ i2 k =1 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 19 Non-linear Regression • Suppose you know that y is related to a function of x in such a way that the predicted values have a non-linear dependence on w, e.g: y=3 xi yi ½ ½ y=2 1 2.5 2 3 y=1 3 2 y=0 3 3 x=0 x=1 x=2 x=3 MLE yi ~ N ( w + xi , σ ) the w? 2 at’s f W h at e o Assume stim e Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 20
11. 11. Non-linear MLE estimation argmax log p( y , y ,..., y | x1 , x2 ,..., xR , σ , w) = 1 2 R w Assuming i.i.d. and argmin ∑ (y ) R then plugging in 2 − w + xi = equation for Gaussian i and simplifying. i =1 w   y − w + xi Setting dLL/dw R  w such that ∑ i = 0 = equal to zero   w + xi   i =1 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 21 Non-linear MLE estimation argmax log p( y , y ,..., y | x1 , x2 ,..., xR , σ , w) = 1 2 R w Assuming i.i.d. and argmin ∑ (y ) R then plugging in 2 − w + xi = equation for Gaussian i and simplifying. i =1 w   y − w + xi Setting dLL/dw R  w such that ∑ i = 0 = equal to zero   w + xi   i =1 We’re down the algebraic toilet t wha ess u So g e do? w Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 22
12. 12. Non-linear MLE estimation argmax log p( y , y ,..., y | x1 , x2 ,..., xR , σ , w) = 1 2 R w Assuming i.i.d. and argmin ∑ ( ) R Common (but not only) approach: then plugging in 2 yi − w + xi = equation for Gaussian Numerical Solutions: and simplifying. i =1 w • Line Search • Simulated Annealing   yi − w + xi Setting dLL/dw R ∑  w such = 0 = • Gradient Descent that equal to zero   w + xi   • Conjugate Gradient i =1 • Levenberg Marquart We’re down the • Newton’s Method algebraic toilet Also, special purpose statistical- t wha optimization-specific tricks such as uess ? So g e do E.M. (See Gaussian Mixtures lecture for introduction) w Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 23 GRADIENT DESCENT f(w): ℜ → ℜ Suppose we have a scalar function We want to find a local minimum. Assume our current weight is w ∂ f (w) GRADIENT DESCENT RULE: w ← w −η ∂w η is called the LEARNING RATE. A small positive number, e.g. η = 0.05 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 24
13. 13. GRADIENT DESCENT f(w): ℜ → ℜ Suppose we have a scalar function We want to find a local minimum. Assume our current weight is w ∂ f (w) GRADIENT DESCENT RULE: w ← w −η ∂w Recall Andrew’s favorite default value for anything η is called the LEARNING RATE. A small positive number, e.g. η = 0.05 QUESTION: Justify the Gradient Descent Rule Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 25 Gradient Descent in “m” Dimensions f(w ) : ℜ m → ℜ Given ∂  f (w )    ∂w1   points in direction of steepest ascent. ∇f (w) =  M ∂   ∂w f (w) m  ∇f (w) is the gradient in that direction w ← w -η∇f (w) GRADIENT DESCENT RULE: Equivalently ∂ f (w ) ….where wj is the jth weight wj ← wj - η ∂w j “just like a linear feedback system” Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 26
14. 14. What’s all this got to do with Neural Nets, then, eh?? For supervised learning, neural nets are also models with vectors of w parameters in them. They are now called weights. As before, we want to compute the weights to minimize sum- of-squared residuals. Which turns out, under “Gaussian i.i.d noise” assumption to be max. likelihood. Instead of explicitly solving for max. likelihood weights, we use GRADIENT DESCENT to SEARCH for them. our eyes. ression in y s exp , a querulou y?” you ask “Wh e later.” ply: “We’ll se “Aha!!” I re Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 27 Linear Perceptrons They are multivariate linear models: Out(x) = wTx And “training” consists of minimizing sum-of-squared residuals by gradient descent. ∑ (Out (x ) − )2 Ε= yk k k ∑ (w ) 2 Τ = x k − yk k QUESTION: Derive the perceptron training rule. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 28
15. 15. Linear Perceptron Training Rule R E = ∑ ( yk − w T x k ) 2 k =1 Gradient descent tells us we should update w thusly if we wish to minimize E: ∂E wj ← wj - η ∂w j ∂E ? So what’s ∂w j Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 29 Linear Perceptron Training Rule R ∂E ∂ R E = ∑ ( yk − w T x k ) 2 =∑ ( yk − w T x k ) 2 ∂w j k =1 ∂w j k =1 ∂ R = ∑ 2( yk − w T x k ) Gradient descent tells us ( yk − w T x k ) ∂w j we should update w k =1 thusly if we wish to ∂ R = −2∑ δk minimize E: wT xk ∂w j k =1 ∂E …where… wj ← wj - η δk = yk − w T x k ∂w j ∂ R m = −2∑ δk ∑w x i ki ∂w j k =1 i =1 ∂E ? So what’s R ∂w j = −2∑ δk xkj k =1 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 30
16. 16. Linear Perceptron Training Rule R E = ∑ ( yk − w T x k ) 2 k =1 Gradient descent tells us we should update w thusly if we wish to minimize E: ∂E wj ← wj - η R w j ← w j + 2η∑ δk xkj ∂w j …where… k =1 ∂E R = −2∑ δk xkj ∂w j We frequently neglect the 2 (meaning k =1 we halve the learning rate) Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 31 The “Batch” perceptron algorithm 1) Randomly initialize weights w1 w2 … wm 2) Get your dataset (append 1’s to the inputs if you don’t want to go through the origin). 3) for i = 1 to R δ i := yi − wΤxi 4) for j = 1 to m R w j ← w j + η ∑ δ i xij i =1 5) if ∑ δ i 2 stops improving then stop. Else loop back to 3. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 32
17. 17. δ i ← yi − w Τ x i A RULE KNOWN BY w j ← w j + ηδ i xij MANY NAMES rule off e wH Rul idro LMS W The e Th The delta ru le Th e ad alin er ule Classical conditioning Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 33 If data is voluminous and arrives fast Input-output pairs (x,y) come streaming in very quickly. THEN Don’t bother remembering old ones. Just keep using new ones. observe (x,y) δ ← y − wΤx ∀j w j ← w j + η δ x j Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 34
19. 19. Gradient Descent vs Matrix Inversion for Linear Perceptrons GD Advantages (MI disadvantages): • Biologically plausible • With very very many attributes each iteration costs only O(mR). If fewer than m iterations needed we’ve beaten Matrix Inversion But we’ll • More easily parallelizable (or implementable in wetware)? GD Disadvantages (MIsoon see that advantages): GD • It’s moronic has an important extra • It’s essentially a slow implementation of a way to build the XTX matrix and then solve a set of linear equations trick up its sleeve • If m is small it’s especially outageous. If m is large then the direct matrix inversion method gets fiddly but not impossible if you want to be efficient. • Hard to choose a good learning rate • Matrix inversion takes predictable time. You can’t be sure when gradient descent will stop. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 37 Perceptrons for Classification What if all outputs are 0’s or 1’s ? or We can do a linear fit. Our prediction is 0 if out(x)≤1/2 1 if out(x)>1/2 WHAT’S THE BIG PROBLEM WITH THIS??? Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 38
20. 20. Perceptrons for Classification What if all outputs are 0’s or 1’s ? or Blue = Out(x) We can do a linear fit. Our prediction is 0 if out(x)≤½ 1 if out(x)>½ WHAT’S THE BIG PROBLEM WITH THIS??? Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 39 Perceptrons for Classification What if all outputs are 0’s or 1’s ? or Blue = Out(x) We can do a linear fit. Our prediction is 0 if out(x)≤½ Green = Classification 1 if out(x)>½ Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 40
21. 21. Classification with Perceptrons I ∑ (y )2 − w Τxi . Don’t minimize i Minimize number of misclassifications instead. [Assume outputs are ∑ (y ( )) +1 & -1, not +1 & 0] − Round w Τ x i i where Round(x) = -1 if x<0 NOTE: CUTE & NON OBVIOUS WHY 1 if x≥0 THIS WORKS!! The gradient descent rule can be changed to: if (xi,yi) correctly classed, don’t change w w - xi if wrongly predicted as 1 w w + xi if wrongly predicted as -1 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 41 Classification with Perceptrons II: Sigmoid Functions Least squares fit useless This fit would classify much better. But not a least squares fit. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 42
22. 22. Classification with Perceptrons II: Sigmoid Functions Least squares fit useless This fit would classify much SOLUTION: better. But not a least squares fit. Instead of Out(x) = wTx We’ll use Out(x) = g(wTx) where g( x) : ℜ → (0,1) is a squashing function Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 43 The Sigmoid 1 g ( h) = 1 + exp(− h) Note that if you rotate this curve through 180o centered on (0,1/2) you get the same curve. i.e. g(h)=1-g(-h) Can you prove this? Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 44
23. 23. The Sigmoid 1 g ( h) = 1 + exp(− h) Now we choose w to minimize ∑ [ yi − Out(x i )] = ∑ [yi − g (w Τ x i )] R R 2 2 i =1 i =1 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 45 Linear Perceptron Classification Regions 0 0 0 1 X2 1 1 X1 Out(x) = g(wT(x,1)) We’ll use the model = g(w1x1 + w2x2 + w0) Which region of above diagram classified with +1, and which with 0 ?? Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 46
24. 24. Gradient descent with sigmoid on a perceptron First, notice g ' ( x ) = g ( x )(1 − g ( x )) − e− x 1 Because : g ( x ) = so g ' ( x ) = 1 + e− x 2 1 + e − x      1 −1 − e− x −1  1 1 1  = − g ( x )(1 − g ( x )) 1 − = = − = 2 1 + e− x 1 + e− x  1 + e− x  2   1 + e − x  1 + e − x            Out(x) = g  ∑ wk xk  The sigmoid perceptron k  update rule: 2    Ε = ∑  yi − g  ∑ wk xik     k  i R w j ← w j + η ∑ δ i gi (1 − gi )xij   ∂    ∂Ε  = ∑ 2 yi − g  ∑ wk xik   − g  ∑ wk xik       ∂w j  k  ∂w j k    i =1 i m      ∂ = ∑ − 2 yi − g  ∑ wk xik  g '  ∑ wk xik  ∑w x gi = g  ∑ w j xij    k ik where  ∂w j k   k    i k  j =1  = ∑ − 2δ i g (net i )(1 − g (net i ))xij δ i = yi − gi i net i = ∑ wk xk where δ i = yi − Out(x i ) k Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 47 Other Things about Perceptrons • Invented and popularized by Rosenblatt (1962) • Even with sigmoid nonlinearity, correct convergence is guaranteed • Stable behavior for overconstrained and underconstrained problems Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 48
25. 25. Perceptrons and Boolean Functions If inputs are all 0’s and 1’s and outputs are all 0’s and 1’s… X2 • Can learn the function x1 ∧ x2 X1 X2 • Can learn the function x1 ∨ x2 . X1 • Can learn any conjunction of literals, e.g. x1 ∧ ~x2 ∧ ~x3 ∧ x4 ∧ x5 QUESTION: WHY? Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 49 Perceptrons and Boolean Functions • Can learn any disjunction of literals e.g. x1 ∧ ~x2 ∧ ~x3 ∧ x4 ∧ x5 • Can learn majority function f(x1,x2 … xn) = 1 if n/2 xi’s or more are = 1 0 if less than n/2 xi’s are = 1 • What about the exclusive or function? f(x1,x2) = x1 ∀ x2 = (x1 ∧ ~x2) ∨ (~ x1 ∧ x2) Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 50
26. 26. Multilayer Networks The class of functions representable by perceptrons is limited   ( ) Out(x) = g w Τ x = g  ∑ w j x j    j  Use a wider representation !    Out(x) = g  ∑W j g  ∑ w jk x jk   This is a nonlinear function   k  j Of a linear combination Of non linear functions Of linear combinations of inputs Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 51 A 1-HIDDEN LAYER NET NINPUTS = 2 NHIDDEN = 3  NINS  v1 = g ∑ w1k xk    w11  k =1  w1 x1 w21  NINS  w31 v2 = g ∑ w2k xk   N HID  w2 Out = g  ∑ Wk vk       k =1  w12  k =1  w22 x2 w3  NINS  w32 v3 = g ∑ w3k xk     k =1  Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 52
27. 27. OTHER NEURAL NETS 1 x1 x2 x3 2-Hidden layers + Constant Term “JUMP” CONNECTIONS x1  N INS  N HID Out = g  ∑ w0 k xk + ∑Wk vk  x2    k =1  k =1 Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 53 Backpropagation    Out(x) = g  ∑W j g  ∑ w jk xk     k  j Find a set of weights {W j },{w jk } to minimize ∑ ( y − Out(x )) 2 i i i by gradient descent. That’s it! That’s it! That’s the backpropagation That’s the backpropagation algorithm. algorithm. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 54
28. 28. Backpropagation Convergence Convergence to a global minimum is not guaranteed. •In practice, this is not a problem, apparently. Tweaking to find the right number of hidden units, or a useful learning rate η, is more hassle, apparently. IMPLEMENTING BACKPROP: Differentiate Monster sum-square residual Write down the Gradient Descent Rule It turns out to be easier & computationally efficient to use lots of local variables with names like hj ok vj neti etc… Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 55 Choosing the learning rate • This is a subtle art. • Too small: can take days instead of minutes to converge • Too large: diverges (MSE gets larger and larger while the weights increase and usually oscillate) • Sometimes the “just right” value is hard to find. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 56
29. 29. Learning-rate problems From J. Hertz, A. Krogh, and R. G. Palmer. Introduction to the Theory of Neural Computation. Addison-Wesley, 1994. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 57 Improving Simple Gradient Descent Momentum Don’t just change weights according to the current datapoint. Re-use changes from earlier iterations. Let ∆w(t) = weight changes at time t. Let be the change we would make with ∂Ε −η ∂w regular gradient descent. Instead we use ∂Ε ∆w (t +1) = −η + α∆w (t ) ∂w w (t + 1) = w (t ) + ∆w (t ) Momentum damps oscillations. momentum parameter A hack? Well, maybe. Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 58
30. 30. Momentum illustration Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 59 Improving Simple Gradient Descent Newton’s method ∂E 1 T ∂ 2 E E ( w + h) = E ( w ) + hT +h h + O(| h |3 ) ∂w 2 ∂w 2 If we neglect the O(h3) terms, this is a quadratic form Quadratic form fun facts: If y = c + bT x - 1/2 xT A x And if A is SPD Then xopt = A-1b is the value of x that maximizes y Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 60