Successfully reported this slideshow.
Upcoming SlideShare
×

# Quadratic form and functional optimization

1,250 views

Published on

This slideshow describes a mathematics topic on quadratic form in English.
I made the slideshow to understand this topic on a deeper way.ใ
I like linear algebra very much!!

Published in: Education
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

### Quadratic form and functional optimization

1. 1. Quadratic Form and Functional Optimization 9th June, 2011 Junpei Tsuji
2. 2. Optimization of multivariate quadratic function ๐ฅ1 1 3 1 ๐ฅ1๐ฝ ๐ฅ1 , ๐ฅ2 = 1.2 + 0.2, 0.3 ๐ฅ2 + ๐ฅ1 , ๐ฅ2 ๐ฅ2 2 1 4 3 2 = 1.2 + 0.2๐ฅ1 + 0.3๐ฅ2 + ๐ฅ1 + ๐ฅ1 ๐ฅ2 + 2๐ฅ2 2 2 ๐ฅ1 , ๐ฅ2 , ๐ฝ = 0.045, 0.064, 1.1881
3. 3. Quadratic approximation 1 ๐ ๐ โ ๐ ฬ + ๐ฑฬ โ ๐ โ ๏ฟฝ + ๐ ๐โ๏ฟฝ ๐ ๐๏ฟฝ ๐ฏ ๐โ๏ฟฝ ๐By Taylors expansion 2 constant linear form quadratic form ๐ โถ= ๐ฅ1 , ๐ฅ2 , โฏ ๐ฅ ๐ ๐where ๐ ฬ โถ= ๐ ๏ฟฝ ๐โข ๐ฑฬ: = , ,โฏ, ๐๐ ๐๐ ๐๐โข ๐๐ฅ1 ๐๐ฅ2 ๐๐ฅ ๐ ๏ฟฝ ๐=๐โข Jacobian (gradient) โฏ ๐2 ๐ ๐2 ๐ ๐๐ฅ1 ๐๐ฅ1 ๐๐ฅ1 ๐๐ฅ ๐โข ๏ฟฝ โถ= ๐ฏ โฎ โฑ โฎ โฏ ๐2 ๐ ๐2 ๐ Hessian (constant) ๐๐ฅ ๐ ๐๐ฅ1 ๐๐ฅ ๐ ๐๐ฅ ๐ ๏ฟฝ ๐=๐
4. 4. Completing the square 1 ๐ ๐ = ๐ ฬ + ๐ฑฬ โ ๐ โ ๏ฟฝ + ๐ ๐โ๏ฟฝ ๐ ๐๏ฟฝ ๐ฏ ๐โ๏ฟฝ ๐ 2โข Let ๏ฟฝ = ๐โ where ๐ฑ ๐โ ๐ = ๐ then ๐ 1 ๐ ๐ = ๐โ + ๐ โ ๐โ ๐ ๐ฏโ ๐ โ ๐โ 2 constant quadratic form
5. 5. Completing the square 1 ๐ ๐ ๐ = ๐ + ๐ ๐ + ๐ ๐จ๐จ ๐ 2 1 ๐ ๐ = ๐+ ๐ โ ๐0 ๐ ๐จ ๐ โ ๐0 2 1 1 1 = ๐ + ๐0 ๐ ๐จ๐0 โ ๐0 ๐ ๐จ + ๐จ ๐ ๐ + ๐ ๐ ๐จ๐จ 2 2 2 ๐๐ =โ ๐0 ๐ ๐จ + ๐จ ๐ 1 2 ๐0 ๐ = โ2๐ ๐ ๐จ + ๐จ ๐ โ1โข ๐0 = โ2 ๐จ + ๐จ ๐ โ1 ๐ ๐= ๐+ ๐0 ๐ ๐จ๐0 1 1 ๐ 2 ๐= ๐โ ๐0 ๐จ๐0 = ๐ โ 2๐ ๐ ๐จ + ๐จ ๐ ๐จ ๐จ+ ๐จ๐ ๐โข โ1 โ1 2 ๐ ๐ = ๐ โ 2๐ ๐ ๐จ + ๐จ ๐ โ1 ๐จ ๐จ+ ๐จ๐ โ1 ๐Therefore, 1 + ๐ + 2 ๐จ + ๐จ ๐ โ1 ๐ ๐ ๐จ ๐ + 2 ๐จ + ๐จ ๐ โ1 ๐ 2 If ๐จ was symmetric matrix, 1 ๐ โ1 1 ๐ ๐ = ๐โ ๐ ๐จ ๐+ ๐ + ๐จโ1 ๐ ๐ ๐จ ๐ + ๐จโ1 ๐โข 2 2
6. 6. Quadratic form ๐ ๐๐ = ๐๐ ๐ ๐บ๐บ๐บโข ๐บ is symmetric matrix.where
7. 7. Symmetric matrixโข Symmetric matrix ๐บ is defined as a matrix that satisfies the ๐บ๐ = ๐บ following formula:โข Symmetric matrix ๐บ has real eigenvalues ๐ ๐ and eigenvectors ๐ ๐ that consist of normal orthogonal base. ๐บ๐ ๐ = ๐ ๐ ๐ ๐where ๐1 โฅ ๐2 โฅ โฏ โฅ ๐ ๐ ๐ ๐ , ๐ ๐ = ๐ฟ ๐๐ ๐ฟ ๐๐ is Kroneckers delta
8. 8. Diagonalization of symmetric matrixโข We define an orthogonal matrix ๐ผ as follows: ๐ผ = ๐1 , ๐2 , โฏ , ๐ ๐โข Then, ๐ผ satisfies the following formulas: ๐ผ๐ ๐ผ= ๐ฐ โด ๐ผโ1 = ๐ผ ๐โข where ๐ฐ is an identity matrix. ๐บ๐บ = ๐บ ๐1 , ๐2 , โฏ , ๐ ๐ = ๐บ๐1 , ๐บ๐2 , โฏ , ๐บ๐ ๐ ๐1 = ๐1 ๐1 , ๐2 ๐2 , โฏ , ๐ ๐ ๐ ๐ = ๐1 , โฏ , ๐ ๐ โฑ ๐๐ = ๐ผ ๐๐๐๐ ๐1 , ๐2 , โฏ , ๐ ๐ โด ๐บ = ๐ผ ๐๐๐๐ ๐1 , ๐2 , โฏ , ๐ ๐ ๐ผ๐
9. 9. Transformation to principal axis ๐ ๐โฒ = ๐โฒ ๐ ๐บ๐บโฒโข Then, we assume ๐๐ = ๐ผ ๐ ๐, where ๐ = ๐ง1 , ๐ง1 , โฏ , ๐ง ๐ . ๐ ๐ผ ๐ ๐ = ๐ผ ๐ ๐ ๐ ๐บ ๐ผ ๐ ๐ = ๐ ๐ ๐ผ๐บ๐ผ ๐ ๐ = ๐ ๐ ๐๐๐๐ ๐1 , ๐2 , โฏ , ๐ ๐ ๐ ๐ โด ๐ ๐ = ๏ฟฝ ๐ ๐ ๐ง ๐2 ๐=1
10. 10. Contour surfaceโข If we assume ๐ ๐ equals constant ๐, ๐ ๐ ๐ = ๏ฟฝ ๐ ๐ ๐ง ๐2 = ๐ ๐=1โข When ๐ = 2, โ a locus of ๐ illustrates an ellipse if ๐1 ๐2 > 0. โ a locus of ๐ illustrates a hyperbola if ๐1 ๐2 < 0.
11. 11. Contour surface ๐ง2 2 ๐ ๐ = ๏ฟฝ ๐ ๐ ๐ง ๐ 2 = ๐๐๐๐๐. ๐=1 ๐1 ๐2 > 0 ๐ง1maximal or minimal point ๐ ๐ฅ1 , ๐ฅ2 = โ๐ฅ1 2 โ 2๐ฅ2 2 + 20.0
12. 12. Transformation to principal axis ๐ฅ๐ฅ2 ๐ ๐๐ = ๐๐๐๐๐. ๐ฅ๐ฅ1 ๐๐ = ๐ผ ๐ ๐ โด ๐ = ๐ผ๐โฒ Transformation to principal axis
13. 13. Parallel translation ๐ฅ๐ฅ2๐ฅ2 ๏ฟฝ ๐ ๐ฅ๐ฅ1 ๐ ๐ = ๐๐๐๐๐. ๐ฅ1 ๐๐ = ๐ โ ๏ฟฝ๐
14. 14. 1Contour surface of quadratic function ๐ ๐ = ๐ + โ ๐ โ ๐โ ๐ ๐ฏโ ๐ โ ๐โ 2 ๐ฅ2 ๏ฟฝ ๐ ๐ ๐ = ๐๐๐๐๐. ๐ฅ1
15. 15. Contour surface ๐ง2 2 ๐ ๐ = ๏ฟฝ ๐ ๐ ๐ง ๐ 2 = ๐๐๐๐๐. ๐=1 ๐1 ๐2 < 0 ๐ง1saddle point ๐ ๐ฅ1 , ๐ฅ2 = ๐ฅ1 2 โ ๐ฅ2 2
16. 16. Stationary points๐ ๐ฅ1 , ๐ฅ2 = ๐ฅ1 3 + ๐ฅ2 3 + 3๐ฅ1 ๐ฅ2 + 2 maximal point saddle point
17. 17. Stationary points 1 3๐ ๐ฅ1 , ๐ฅ2 = exp โ ๐ฅ1 + ๐ฅ1 โ ๐ฅ2 2 3 saddle point maximal point
18. 18. Newton-Raphson method ๐๐ ๐ = ๐ where ๐ ๐ is ๐-th polynomial byโข Newtonโs method is an approximate solver of using a quadratic approximation. ๐ ๐ quadratic approximation of ๐ ๐ in ๐ 1 ๐ ๐ + ฮ๐ โ ๐ ๐ + ๐ฑ ๐ โ ฮ๐ + ฮ๐ ๐ ๐ฏ ๐ ฮ๐ 2 ๐๐ ๐ + ฮ๐ ๐๐ ๐โ = ๐ ๐ ฮ๐ = ๐ฑ ๐ ๐ + ๐ฏ ๐ ฮ๐ ๐โ ๐ + ๐ซ๐ ๐ ๐
19. 19. Algorithm of Newtonโs methodProcedure Newton (๐ฑ ๐ , ๐ฏ ๐ ) 1. Initialize ๐. 2. Calculate ๐ฑ ๐ and ๐ฏ ๐ . equation and giving โ๐ : ๐ฑ ๐ ๐ + ๐ฏ ๐ โ๐ = ๐ 3. Solve the following simultaneous 4. Update ๐ as follows: ๐ โ ๐ + โ๐ 5. If โ๐ < ๐ฟ then return ๐ else go back to 2.
20. 20. Linear regression ๐ ๐ฆ ๐ฆ = ๐ ๐ = ๐ฝ0 + ๏ฟฝ ๐ฝ ๐ ๐ฅ ๐ ๐ samples ๐ ๐, ๐ฆ ๐ ๐=1 ๐ ๐-th dimensional spaceWe would like to find ๐ทโ that minimizes the residual sum of square (RSS).
21. 21. Linear regression min RSS ๐ท ๐ท 2 ๐ ๐ ๐โข where RSS ๐ท = ๏ฟฝ ๐ฆ ๐ โ ๐ ๐ ๐ 2 = ๏ฟฝ ๐ฆ๐ โ ๐ฝ0 + ๏ฟฝ ๐ฝ ๐ ๐ฅ ๐๐ ๐=1 ๐=1 ๐=1โข Given ๐ฟ, ๐, ๐ท as follows: ๐ฅ11 โฏ ๐ฅ1๐ 1 ๐ฆ1 ๐ฝ1 ๐ฟ= โฎ โฑ โฎ โฎ , ๐= โฎ , ๐ท= โฎ ๐ฅ ๐๐ โฏ ๐ฅ ๐๐ 1 ๐ฆ๐ ๐ฝ๐ โด RSS ๐ท = ๐ โ ๐ฟ๐ท 2
22. 22. Linear regression RSS ๐ท = ๐ฝ ๐ท = ๐ โ ๐ฟ๐ท 2 = ๐ โ ๐ฟ๐ท ๐ ๐ โ ๐ฟ๐ท = ๐ ๐ ๐ โ ๐ท ๐ ๐ฟ ๐ ๐ โ ๐ ๐ ๐ฟ๐ท + ๐ท ๐ ๐ฟ ๐ ๐ฟ๐ท ๐๐ ๐ท = ๐ ๐ ๐๐ท ๐ท๐ ๐ = ๐โข ๐ ๐๐ท ๐ท ๐ ๐จ๐ท = ๐จโข ๐ ๐๐ท ๐๐ฝ ๐ฝโฒ ๐ท = = โ2๐ฟ ๐ ๐ + 2๐ฟ ๐ ๐ฟ๐ทโข ๐๐ท
23. 23. Linear regressionGiven ๐ทโ that satisfies ๐ฝโฒ ๐ทโ = ๐, ๐ฟ ๐ ๐ = ๐ฟ ๐ ๐ฟ๐ทโ ๐ ๐ ๐ฟ = ๐ทโ ๐ ๐ฟ ๐ ๐ฟ โด ๐ทโ = ๐ฟ๐ ๐ฟ โ1 ๐ฟ๐ ๐ โด ๐ฝ ๐ท = ๐ ๐ โ ๐ท ๐ฟ ๐ฟ๐ท โ ๐ท ๐ฟ ๐ ๐ฟ๐ท + ๐ท ๐ ๐ฟ ๐ ๐ฟ๐ท ๐ ๐ ๐ โ โ๐ โด ๐ฝ ๐ท = ๐ ๐ ๐ โ ๐ทโ ๐ ๐ฟ ๐ ๐ฟ๐ทโ + ๐ทโ ๐ ๐ฟ ๐ ๐ฟ๐ทโ โ ๐ท ๐ ๐ฟ ๐ ๐ฟ๐ทโ โ ๐ทโ ๐ ๐ฟ ๐ ๐ฟ๐ท + ๐ท ๐ ๐ฟ ๐ ๐ฟ๐ท โด ๐ฝ ๐ท = ๐ ๐ ๐ โ ๐ทโ ๐ ๐ฟ ๐ ๐ฟ๐ทโ + ๐ท โ ๐ทโ ๐ ๐ฟ ๐ ๐ฟ ๐ท โ ๐ทโ completing the square
24. 24. Linear regression ๐ฝ ๐ท = ๐ ๐ ๐ โ ๐ทโ ๐ ๐ฟ ๐ ๐ฟ๐ทโ + ๐ท โ ๐ทโ ๐ ๐ฟ ๐ ๐ฟ ๐ท โ ๐ทโ = ๐ โ ๐ฟ๐ทโ 2 + ๐ท โ ๐ทโ ๐ ๐ฟ ๐ ๐ฟ ๐ท โ ๐ทโ 1 = ๐ฝ ๐ท + โ ๐ท โ ๐ท โ ๐ ๐ฏ ๐ท โ ๐ทโ 2Residual sum of squares (RSS) quadratic form ๐ฝ2 ๐ฝ ๐ท = ๐๐๐๐๐.by Linear Regression ๐ทโ ๐ทโ = ๐ฟ ๐ ๐ฟ โ1 ๐ฟ ๐ ๐ ๐ฏ = 2๐ฟ ๐ ๐ฟ ๐ฝ1
25. 25. Hessianโข ๐ฏโ = 2๐ฟ ๐ ๐ฟ ๐2 ๐ฝ ๐๐ฝ ๐ ๐๐ฝ ๐โข ๐ฏ has the following two features: ๐ฏ๐ = ๐ฏ โ ๐ โ  ๐, ๐ ๐ ๐ฏ๐ฏ > 0 โ symmetric matrix: โ positive-definite matrix:Therefore, ๐ทโ = ๐ฟ๐ ๐ฟ โ1 ๐ฟ ๐ ๐ is the minimumof ๐ฝ ๐ท .
26. 26. Analysis of residuals ๐โ = ๐ฟ๐ทโโข Then, we substitute ๐ทโ = ๐ฟ ๐ ๐ฟ โ1 ๐ฟ ๐ ๐ in the above, ๐โ = ๐ฟ๐ทโ = ๐ฟ ๐ฟ ๐ ๐ฟ โ1 ๐ฟ๐ ๐ โด ๐โ = โ๐ (Hat matrix)โข the vector of residuals ๐ can be expressed by follows: ๐ = ๐ โ ๐โ = ๐ โ โ๐ = ๐ฐ โ โ ๐ ๐๐๐ ๐ = ๐๐๐ ๐ฐ โ โ ๐ = ๐ฐ โ โ ๐๐๐ ๐ ๐ฐ โ โ ๐
27. 27. Analysis of residuals โ = ๐ฟ ๐ฟ ๐ ๐ฟ โ1 ๐ฟ ๐The hat matrix โ is a projection matrix, which1. Projection: โ 2 = โsatisfies the following equations: โ 2 = โ โ โ = ๐ฟ ๐ฟ ๐ ๐ฟ โ1 ๐ฟ ๐ โ ๐ฟ ๐ฟ ๐ ๐ฟ โ1 ๐ฟ๐ = ๐ฟ ๐ฟ ๐ ๐ฟ โ1 ๐ฟ ๐ ๐ฟ ๐ฟ ๐ ๐ฟ โ1 ๐ฟ ๐ = ๐ฟ ๐ฟ ๐ ๐ฟ โ1 ๐ฟ ๐ = โ2. Orthogonal: โ ๐ = โ
28. 28. Analysis of residuals ๐ฅ11 โฏ ๐ฅ1๐ 1 ๐ฝ1 โ ๐ฆ1 โ โฎ โฎ = โฎ โฑ โฎ โฎ ๐ฝ๐ โ ๐ฆ ๐โ ๐ฅ ๐1 โฏ ๐ฅ ๐๐ 1 ๐ฝ0 โ ๐ฅ11 ๐ฅ1๐ 1= ๐ฝ1 โ โฎ + โฏ + ๐ฝ ๐โ โฎ + ๐ฝ0 โฎ โ ๐ฅ ๐1 ๐ฅ ๐๐ 1 ๐1 ๐๐ ๐ ๐+1 = ๐ linear combination in ๐ + 1 -th vector space
29. 29. Analysis of residuals ๐ ๐โ = โ๐ (Projection) ๐๐ ๐โ ๐๐ ๐ + 1 -th dimensional super surface๐-th dimensional space
30. 30. Analysis of residuals ๐ = ๐ฟ๐ทโข ๐ท = ๐ฟโ1 ๐, where ๐ฟโ1 is M-P generalized inverse. ๐= ๐ ๐> ๐ 1. Unique solution: ๐< ๐ 2. Many solutions: ๐ฟโ1 3. No solution: ๐ฟ โ1 =๏ฟฝ ๐ฟ๐ฟ ๐ฟ๐ฟ๐ฟ โ1 ๐ท = ๐ฟโ1 ๐ is min in ๐ท ๐ฟ๐ฟ๐ฟ โ1 ๐ฟ๐ฟ ๐ โ ๐ฟ๐ท 2 is minโข