Reading the Lindley-Smith 1973 paper on linear Bayes estimators

3,545 views

Published on

Presentation made by Kaniav Kamary at the Reading classics graduate seminar in Paris-Dauphine

Published in: Real Estate
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,545
On SlideShare
0
From Embeds
0
Number of Embeds
2,490
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Reading the Lindley-Smith 1973 paper on linear Bayes estimators

  1. 1. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ............... . . Bayes Estimates for the Linear Model . .. . . Reading Seminar in Statistical Classics Director: C. P. Robert Presenter: Kaniav Kamary 12 Novembre, 2012
  2. 2. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Outline . . . Introduction 1 The Model and the bayesian methods . . . Exchangeability 2 . . . General bayesian linear model 3 . . . Examples 4 . . . Estimation with unknown Covariance 5
  3. 3. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............The Model and the bayesian methodsThe linear model : Structure of the linear model: E(y ) = Aθ y : a vector of the random variables A: a known design Matrix Θ: unknown parameters For estimating Θ: The usual estimate by the method of least squares. Unsatisfactory or inadmissibility in demensions greater than two. Improved estimates with knowing prior information about the parameters in the bayesian framework
  4. 4. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Outline . . . Introduction 1 . . . Exchangeability 2 un example . . . General bayesian linear model 3 . . . Examples 4 . . . Estimation with unknown Covariance 5
  5. 5. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............un exampleThe concept of exchangeability In general linear model suppose A = I : E(yi ) = Θi for i = 1, 2, . . . , n and yi ∼ N(θi , σ 2 ) iid The distribution of θi is exchangeable if: The prior opinion of θi is the same of that of θj or any other θk where i, j, k = 1, 2, . . . , n. In the other hand: A sequence θ1 , . . . , θn of random variables is said to be exchangeable if for all k = 2, 3, . . . θ1 , . . . , θn ∼ θπ(1) , θπ(2) , θπ(k) = for all π ∈ S(k ) where S(k ) is the group of permutation of 1, 2, . . . , k
  6. 6. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............un exampleThe concept of exchangeability. . . One way for obtaining an exchangeable distribution p(Θ): ∏ n p(Θ) = p(θi | µ)dQ(µ) (1) i=1 p(Θ): exchangeable prior knowledge described by a mixture Q(µ): arbitrary probability distribution for each µ µ: the hyperparameters A linear structure to the parameters: E(θi ) = µ
  7. 7. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............un exampleEstimate of Θ If θi ∼ N(µ, τ 2 ): a closer parallelism between the two stage for y and Θ By assuming that µ have a uniform distribution over the real line then: yi y. 2 + τ2 θi∗ = σ 1 1 (2) σ 2 + τ2 ∑n i=1 yi where y. = n and θi∗ = E(θi | y ).
  8. 8. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............un exampleThe features of θi∗ ˆ A weighted averages of yi = θi , overall mean y. and inversely proportional to the variances of yi and θi A biased estimate of θi Use the estimates of τ 2 and σ 2 An admissible estimate with known σ 2 , τ 2 A bayes estimates as substitution for the usual least-squares estimates
  9. 9. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............un exampleThe features of θi∗ . . . . Judging the merit of θi with one of the other estimates . .. The condition that the average M.S.E for θi ∗ to be less than that ˆ for θi is: ∑ (θi − θ. )2 < 2τ 2 + σ 2 (3) n−1 ∑ s2 = (θi −θ. ) is an usual estimate for τ 2 . Hence, the chance of 2 n−1 unequal (3) being satisfied is high for n as law as 4 and rapidly tends to 1 as n increases. . .. . .
  10. 10. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Outline . . . Introduction 1 . . . Exchangeability 2 . . . General bayesian linear model 3 The posterior distribution of the parameters . . . Examples 4 . . . Estimation with unknown Covariance 5
  11. 11. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............The posterior distribution of the parametersThe structure of the model Let: Y : a column vector . Lemma . .. Suppose Y ∼ N(A1 Θ1 , C1 ) and Θ1 ∼ N(A2 Θ2 , C2 ) that Θ1 is a vector of P1 parameters, that Θ2 is a vector of P2 hyperparameters. Then (a): Y ∼ N(A1 A2 Θ2 , C1 + A1 C2 AT ), 1 and (b): Θ1 | Y ∼ N(Bb, B) where: −1 −1 B −1 = AT C1 A1 + C2 1 −1 −1 . b = AT C1 y + C2 A2 Θ2 1 (4) .. . .
  12. 12. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............The posterior distribution of the parametersThe posterior distribution with three stages . Theorem . .. With the assumptions of the Lemma, suppose that given Θ3 , Θ2 ∼ N(A3 Θ3 , C3 ) then for i = 1, 2, 3: Θ1 | {Ai }, {Ci }, Θ3 , Y ∼ N(Dd, D) with −1 −1 D −1 = AT C1 A1 + {C2 + A2 C3 AT } 1 2 (5) and T −1 T −1 −1 . d = A1 C1 y + A1 C1 A1 + {C2 + A2 C3 A2 } T A2 A3 Θ3 (6) .. . .
  13. 13. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............The posterior distribution of the parametersThe properties . Result of the Lemma . .. For any matrices A1 , A2 , C1 and C2 of appropriate dimensions and for witch the inverses stated, we have: −1 −1 −1 −1 −1 C1 − C1 A1 (AT C1 A1 + C2 )−1 AT C1 = (C1 + A1 C2 AT ) 1 1 1 . (7) .. . . . Properties of the bayesian estimation . .. The E(Θ1 | {Ai }, {Ci }, Θ3 , Y ) is: A weighed average of the least-squares estimates −1 −1 −1 (AT C1 A1 ) AT C1 y . 1 1 A weithed average of the prior mean A2 A3 Θ3 . It may be regarded as a point estimate of Θ1 to replace the usual least-squares estimate. . .. . .
  14. 14. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............The posterior distribution of the parametersResults of the Theorem . Corollary1 . .. An alternative expression for D −1 : T −1 −1 −1 T −1 −1 T −1 −1 . A1 C1 A1 + C2 − C2 A2 {A2 C2 A2 + C3 } A2 C2 (8) .. . . . Corollary2 . .. −1 If C3 = 0, the posterior distribution of Θ1 is N(D0 d0 , D0 ) with: −1 −1 −1 −1 −1 −1 T −1 D0 = AT C1 A1 + C2 − C2 A2 {AT C2 A2 } 1 2 A2 C2 (9) and −1 . d0 = AT C1 y 1 (10) .. . .
  15. 15. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Outline . . . Introduction 1 . . . Exchangeability 2 . . . General bayesian linear model 3 . . . Examples 4 Two-factor Experimental Designs Exchangeability Between Multiple Regression Equation Exchangeability within Multiple Regression Equation . . . Estimation with unknown Covariance 5
  16. 16. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Two-factor Experimental DesignsThe structure of the Two-factor Experimental Designs The usual model of n observations with the errors independent N(0, σ 2 ): E(yij ) = µ + αi + βj , 1 ≤ i ≤ t, 1 ≤ j ≤ b ΘT 1 = (µ, α1 , . . . , αt , β1 , . . . , βb ) (11) yij : an observation in the ith treatment and the jth block. The exchangeable prior knowledge of {αi } and {βj } but independent αi ∼ N(0, σα ), βj ∼ N(0, σβ ), µ ∼ N(w, σµ ) 2 2 2 The vague prior knowledge of µ and σµ → ∞ 2 −1 C2 : the diagonal matrix that leading diagonal of C2 is −2 −2 −2 −2 (0, σα , . . . , σα , σβ , . . . , σβ ) C1 : the unit matrix times σ 2
  17. 17. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Two-factor Experimental DesignsBayesian estimate of the parameters With substituting the assumptions stated and C3 = 0 in to (5) and (6), then: −1 D −1 = σ −2 AT A1 + C2 1 d = σ −2 AT y 1 (12) Hence Θ∗ , the bayes estimate Dd, satisfies the equation as 1 following −1 (AT A1 + σ 2 C2 )Θ∗ = AT y 1 1 1 (13) by solving (13), µ = y.. −1 αi∗ = (bσα + σ 2 ) 2 bσα (yi. − y.. ) 2 −1 βj∗ = (tσβ + σ 2 ) 2 tσβ (y.j − y.. ) 2 (14)
  18. 18. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exchangeability Between Multiple Regression EquationThe structure of the Multiple Regression Equation The usual model for p regressor variables where j = 1, 2, . . . , m: yj ∼ N(Xj βj , Inj σj2 ) (15) A1 : a diagonal matrix with xj as the jth diagonal submatrix ΘT = (β1 , β2 , . . . , βm ) 1 T T T Suppose variables X and Y were related with the usual linear regression structure and βj ∼ N(ξ, Σ), Θ2 = ξ A2 : a matrix of order mp × p, all of whose p × p submatrices are unit matrices
  19. 19. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exchangeability Between Multiple Regression EquationBayesian estimation for the parameters of the Multiple RegressionEquation. . . The equation for the bayes estimates βj∗ is   σ1 −2 X1 T X1 + Σ−1 ··· 0  . .   . . σ2 −2 X2 T X2 + Σ−1 . .  0 ··· σm −2 Xm T Xm + Σ−1       β1 ∗ β. ∗ σ1 −2 X1 T y  β2 ∗   β. ∗   σ2 −2 X1 T y    −1     × . −Σ  . = .  (16)  . .   . .   . .  βm ∗ β. ∗ σm −2 X1 T y ∑ βi ∗ where β. ∗ = m .
  20. 20. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exchangeability Between Multiple Regression EquationBayesian estimation for the parameters of the Multiple RegressionEquation By solving equation (16) for βj ∗ , the bayes estimate is −1 βj ∗ = (σj −2 Xj T Xj + Σ−1 ) (σj −2 Xj T y + Σ−1 β. ∗ ) (17) Noting that D0 −1 , given in Corollary 2 (9) and the matrix Lemma 7, we obtain a weighted form of (17) with β. ∗ replaced ∑ by wj βj ∗ : −1 ∑m −1 −1 wi = { (σj −2 Xj T Xj + Σ−1 ) σj −2 Xj T Xj } (σi −2 Xi T Xi + Σ−1 ) σi −2 Xi T Xi j=1 (18)
  21. 21. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exchangeability within Multiple Regression Equationthe model and bayes estimates of the parameters A single multiple regression: y ∼ N(X β, In σ 2 ) (19) The individual regression coefficients in β T = (β1 , β2 , . . . , βp ) are exchangeable and βj ∼ N(ξ, σβ 2 ). . bayes estimate with two possibilities . .. σ2 to suppose vague prior knowledge for ξ with k = σ2 β β ∗ = {Ip + k (X T X )−1 (Ip − p−1 )}−1 β ˆ (20) to put ξ = 0, reflecting a feeling that the βi are small β ∗ = {Ip + k (X T X )−1 }−1 β ˆ (21) . .. . .
  22. 22. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Outline . . . Introduction 1 . . . Exchangeability 2 . . . General bayesian linear model 3 . . . Examples 4 . . . Estimation with unknown Covariance 5 Exposition and method Two-factor Experimental Designs(unknown Covariance) Exch between Multiple Regression(unknown Covariance) Exch within Multiple Regression(unknown Covariance)
  23. 23. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exposition and methodMethod θ: the parameters of interest in the general model ϕ: the nuisance parameters Ci : the unknown dispersion matrices . The method and its defect . .. assign a joint prior distribution to θ and ϕ provide the joint posterior distribution p(θ, ϕ | y ) integrating the joint posterior with respect to ϕ and leaving the posterior for θ for using loss function, necessity another integration for calculate the mean require the constant of proportionality in bayes’s formula for calculating the mean the . above argument is technically most complex to execute. .. . .
  24. 24. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exposition and methodSolution For simplified the method: considering an approximation using the mode of the posterior distribution in place of the mean using the mode of the joint distribution rather than that of the θ-margin taking the estimates derived in section 2 and replace the unknown values of the nuisances parameters by their modal estimates
  25. 25. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exposition and methodSolution. . . The modal value: ∂ ∂ p(θ, ϕ | y ) = 0, p(θ, ϕ | y ) = 0 ∂θ ∂ϕ assuming that p(ϕ | y ) ̸= 0 as ∂ p(θ | y , ϕ) = 0 (22) ∂θ The approximation is good if: the samples are large the resulting posterior distributions approximately normal
  26. 26. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Two-factor Experimental Designs(unknown Covariance) The prior distributions for σ 2 , σα 2 and σβ 2 are invers-χ2 . νλ να λα νβ λ β 2 ∼ χν 2 , 2 ∼ χνα 2 , ∼ χνβ 2 σ σα σβ 2 With assuming the three variances independent. The joint distribution of all quantities: −1 −1 (σ 2 ) 2 (n+ν+2) × exp {νλ + S 2 (µ, α, β)} 2σ 2 −1 −1 ∑2 ×(σα ) 2 2 (t+να +2) exp 2σ2 {να λα + αi } 2 −1 −1 α ∑2 ×(σβ ) 2 (b+νβ +2) exp 2σ2 {νβ λβ + βj } β
  27. 27. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Two-factor Experimental Designs(unknown Covariance)Estimates of the parameters of the model To find the modal estimates: reversing the roles of θ and ϕ with supposing µ, α and β known {νλ + S 2 (µ∗ , α∗ , β ∗ )} s2 = (n + ν + 2) ∑ {να λα + αi ∗2 } sα 2 = (t + να + 2) ∑ {νβ λβ + βj ∗2 } sβ 2 = (23) (b + νβ + 2) solving (13) with trial value of σ 2 , σα and σβ 2 2 inserting the value in to (23)
  28. 28. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch between Multiple Regression(unknown Covariance)Suppositions of the model. . . In the model (15), suppose σj 2 = σ 2 with νλ ∼ χν 2 and Σ−1 has σ2 a Wishart distribution with ρ degree of freedom and matrix R independent of σ 2 . The joint distribution of all the quantities: −1 ∑ m −1n (σ 2 ) 2 × exp{ (yj − Xj βj )T (yj − Xj βj )} 2σ 2 j=1 −1 ∑ m −1m × (| Σ |) 2 exp{ (βj − ξ)T Σ−1 (βj − ξ)} 2 j=1 −1 −1(ρ−p−1) × (| Σ |) 2 tr Σ−1 R} exp{ 2 −1(ν+2) −νλ × (σ 2 ) 2 exp{ 2 } (24) 2σ
  29. 29. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch between Multiple Regression(unknown Covariance)The joint posterior distribution The joint posterior density for β, σ 2 and Σ−1 : −1 ∑ m −1(n+ν+2) T (σ )2 2 × exp{ 2 {m−1 νλ + (yj − Xj βj ) (yj − Xj βj )}} 2σ j=1 −1(m+ρ−p−2) × (| Σ |) 2 −1 ∑ m × exp{ tr Σ−1 {R + (βj − β. )(βj − β. )T }} (25) 2 j=1 ∑m where β. = m−1 j=1 βj .
  30. 30. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch between Multiple Regression(unknown Covariance)The modal estimates . The estimates of the parameters . .. ∑m −1 ∗ T ∗ 2 j=1 {m νλ + (yj − Xj βj ) (yj − Xj βj )} s = (26) (n + ν + 2) and ∑m {R + ∗ − β.∗ )(βj∗ − β.∗ )T } ∗ j=1 (βj Σ = (27) . (m + ρ − p − 2) .. . .
  31. 31. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch between Multiple Regression(unknown Covariance)The modal estimates . . . The posterior distribution of the βj ’s, free of σ 2 and Σ: ∑ n 1 { {m−1 νλ + (yj − Xj βj )T (yj − Xj βj )}}− 2 (n+ν) j=1 − 1 (m+ρ−1) ∑ m 2 ×| R + (βj − β. )(βj − β. )T | (28) j=1 The mode of this distribution can be used in place of the modal values for the wider distribution.
  32. 32. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch between Multiple Regression(unknown Covariance)Application . an application in an educational context . .. data from the American Collage Testing Program 1968, 1969 prediction of grade-point average at 22 collages the results of 4 tests (English, Mathematics, Social Studies, Natural Sciences),p = 5, m = 22, and nj varying from 105 to 739 Table: Comparison of predictive efficiency reduction the error by under 2 per cent by using the bayesian method in the first row but 9 per cent with the quarter sample . .
  33. 33. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch within Multiple Regression(unknown Covariance)Assumptions of the model regression In the model 19 and βj ∼ N(ξ, σβ 2 ), suppose νλ νβ λ β 2 ∼ χν 2 , ∼ χνβ 2 σ σβ 2 The posterior distribution of β, σ 2 and σβ 2 : −1(n+ν+2) −1 (σ 2 ) 2 × exp{ {νλ + (y − X β)T (y − X β)}} 2σ 2 −1(p+νβ +1) × (σβ 2 ) 2 −1 ∑ p × exp{ 2 {νβ λβ + (βj − β. )2 }} (29) 2σβ j=1 ∑p that β. = p−1 j=1 βj .
  34. 34. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch within Multiple Regression(unknown Covariance)The modal estimation. . . The modal equations: −1 β ∗ = {Ip + k ∗ X T X (Ip − p−1 Jp )}−1 β ˆ {νλ + (y − X β ∗ )T (y − X β ∗ )} s2 = (n + ν + 2) ∑p {νβ λβ + j=1 (βj ∗ − β. ∗ )2 } sβ 2 = (30) (p + νβ + 1) where k ∗ = s2 sβ ∗ .
  35. 35. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch within Multiple Regression(unknown Covariance)Comparison between the methods of the estimates The main difference lies in the choice of k in absolute value, the least-squares procedure produce regression estimates too large, of incorrect sign and unstable with respect to small changes in the data The ridge method avoid some of these undesirable features The bayesian method reaches the same conclusion but has the added advantage of dispensing with the rather arbitrary choice of k and allows the data to estimate it Table: 10-factor multiple regression example
  36. 36. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch within Multiple Regression(unknown Covariance)A brief explanation of a recent paper On overview of the Bayesian Linear Model with unknown Variance: Yn×p = Xp×1 + ξ The bayesian approache to fitting the linear model consists of three steps (S.Kuns, 2009)[4]: assign priors to all unknown parameters write down the likelihood of the data given the parameters determine the posterior distribution of the parameters given the data using bayes’ theorem If Y ∼ N(X β, k −1 ) then a conjugate prior distribution for the parameters is: β, k ∼ NG(β0 , Σ0 , a, b). In other word: p−2 −1 f (β, k ) = CK a+ 2 exp k {(β − β0 )T Σ−1 (β − β0 ) + 2b} 0 2 ba where C = p 1 (2π) 2 |Σ0 | 2 Γ(a)
  37. 37. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Exch within Multiple Regression(unknown Covariance)A brief explanation of a recent paper... The posterior distribution is: ∗ + p −1 −1 f (β, k | Y ) ∝ k a 2 exp{ k ((β − β ∗ )T (Σ∗ )−1 (β − β ∗ ) + 2b∗ )} 2 β ∗ = (Σ0 −1 + X T X )−1 ((Σ0 −1 β0 + X T y ) Σ∗ = (Σ0 −1 + X T X )−1 n a∗ = a + 2 1 b∗ = b + (β0 T Σ0 −1 β0 + y T y − (β ∗ )T (Σ∗ )−1 β ∗ ) 2 And β | y follows a multivariate t-distribution: −1 (ν+p) 1 2 f (β | y ) ∝ (1 + (β − β ∗ )T (Σ∗ )−1 (β − β ∗ )) ν
  38. 38. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............References L. D. Brown, On the Admissibility of Invariant Estimators of One or More Location Parameters, The Annals of Mathematical Statistics, Vol. 37, No. 5 (Oct., 1966), pp. 1087-1136. A. E. Hoerl, R. W. kennard, Ridge Regression: Biased Estimation for Nonorthogonal Problems, Technometrics, Vol. 12, No. 1. (Feb., 1970), pp. 55-67. T. Bouche, Formation LaTex, (2007). S. Kunz, The Bayesian Linear Model with unknown Variance, Seminar for Statistics. ETH Zurich, (2009). V. Roy, J. P. Hobert, On Monte Carlo methods for Bayesian multivariate regression models with heavy-tailed errors, (2009).
  39. 39. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............References Thank you for your Attention
  40. 40. Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References . ..... .... ...... ...............Result . Proof. . .. To prove (a), suppose y = A1 Θ1 + u, Θ1 = A2 Θ2 + v that: u ∼ N(0, C1 ) and v ∼ N(0, C2 ) Then y = A1 A2 Θ2 + A1 v + u that: A1 v + u ∼ N(0, C1 + A1 C2 AT ) 1 To prove (b), by using the Bayesian Theorem: 1 p(Θ1 | Y ) ∝ e− 2 Q Q = (Θ1 − Bb)T B −1 (Θ1 − Bb) −1 −1 + y T C1 y + Θ2 T A2 T C2 A2 Θ2 − bT Bb (31) . .

×