Upcoming SlideShare
×

# Lecture 4

913 views

Published on

0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total views
913
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
22
0
Likes
0
Embeds 0
No embeds

No notes for slide

### Lecture 4

1. 1. Outline <ul><li>Regression on a large number of correlated inputs </li></ul><ul><li>A few comments about shrinkage methods, such as ridge regression </li></ul><ul><li>Methods using derived input directions </li></ul><ul><ul><li>Principal components regression </li></ul></ul><ul><ul><li>Partial least squares regression (PLS) </li></ul></ul>
2. 2. Partitioning of the expected squared prediction error <ul><li> bias </li></ul><ul><li>Shrinkage decreases the variance but increases the bias </li></ul><ul><li>Shrinkage methods are more robust to structural changes in the analysed data </li></ul>
3. 3. Advantages of ridge regression over OLS <ul><li>The models are easier to comprehend because strongly correlated inputs tend to get similar regression coefficients </li></ul><ul><li>Generalizations to new data sets are facilitated by a larger robustness to structural changes in the analysed data set </li></ul>
4. 4. Ridge regression - a note on standardization The principal components and the shrinkage in ridge regression are scale-dependent. Inputs are normally standardized to mean zero and variance one prior to the regression
5. 5. Regression methods using derived input directions <ul><li>Extract linear combinations of the inputs as derived features, and then model the target (response) as a linear function of these features </li></ul>x 1 x 2 x p z 1 z 2 z M … … y
6. 6. Absorbance records for ten samples of chopped meat 1 response variable (fat) 100 predictors (absorbance at 100 wavelengths or channels) The predictors are strongly correlated to each other
7. 7. Absorbance records for ten samples of chopped meat High fat samples Low fat samples
8. 8. 3-D plots of absorbance records for samples of meat - channels 1, 50 and 100
9. 9. 3-D plots of absorbance records for samples of meat - channels 40, 50 and 60
10. 10. 3-D plot of absorbance records for samples of meat - channels 49, 50 and 51
11. 11. Matrix plot of absorbance records for samples of meat - channels 1, 50 and 100
12. 12. Principal Component Analysis (PCA) <ul><li>PCA is a technique for reducing the complexity of high dimensional data </li></ul><ul><li>It can be used to approximate high dimensional data with a few dimensions so that important features can be visually examined </li></ul>
13. 13. Principal Component Analysis - two inputs PC1 PC2
14. 14. 3-D plot of artificially generated data - three inputs PC1 PC2
15. 15. Principal Component Analysis <ul><li>The first principal component (PC1) is the direction that maximizes the variance of the projected data </li></ul><ul><li>The second principal component (PC2) is the direction that maximizes the variance of the projected data after the variation along PC1 has been removed </li></ul><ul><li>The third principal component (PC3) is the direction that maximizes the variance of the projected data after the variation along PC1 and PC2 has been removed </li></ul>
16. 16. Eigenvector and eigenvalue <ul><li>In this shear transformation of the Mona Lisa, the picture was deformed in such a way that its central vertical axis (red vector) was not modified, but the diagonal vector (blue) has changed direction. Hence the red vector is an eigenvector of the transformation and the blue vector is not. Since the red vector was neither stretched nor compressed, its eigenvalue is 1. </li></ul>
17. 17. Sample covariance matrix <ul><li>where </li></ul>
18. 18. Eigenvectors of covariance and correlation matrices <ul><li>The eigenvectors of a covariance matrix provide information about the major orthogonal directions of the variation in the inputs </li></ul><ul><li>The eigenvalues provide information about the strength of the variation along the different eigenvectors </li></ul><ul><li>The eigenvectors and eigenvalues of the correlation matrix provide scale-independent information about the variation of the inputs </li></ul>
19. 19. Principal Component Analysis Eigenanalysis of the Covariance Matrix Eigenvalue 2.8162 0.3835 Proportion 0.880 0.120 Cumulative 0.880 1.000 Variable PC1 PC2 X1 0.523 0.852 X2 0.852 -0.523 Loadings
20. 20. Principal Component Analysis Coordinates in the coordinate system determined by the principal components
21. 21. Principal Component Analysis Eigenanalysis of the Covariance Matrix Eigenvalue 1.6502 0.7456 0.0075 Proportion 0.687 0.310 0.003 Cumulative 0.687 0.997 1.000 Variable PC1 PC2 PC3 x 0.887 0.218 -0.407 y 0.034 -0.909 -0.414 z 0.460 -0.354 0.814
22. 22. Scree plot
23. 23. Principal Component Analysis - absorbance data from samples of chopped meat Eigenanalysis of the Covariance Matrix Eigenvalue 26.127 0.239 0.078 0.030 0.002 0.001 0.000 0.000 0.000 Proportion 0.987 0.009 0.003 0.001 0.000 0.000 0.000 0.000 0.000 Cumulative 0.987 0.996 0.999 1.000 1.000 1.000 1.000 1.000 1.000
24. 24. Scree plot - absorbance data One direction is responsible for most of the variation in the inputs