general methods to estimate parameters in linear regression.

1. 1
METHOD OF MOMENTS :
If E(Xi) is some function of , then we can estimate that
function () using only the statistic x
We know how to estimate parameters such as , 2, or p. Now we
need general methods to estimate other parameters
From a sample X1,…,Xn, suppose we wish to estimate some
unknown parameter 
Estimating Parameters

2. 2
Example: X1,…,Xn ~Exponential()
E(X)=1/  (so ()=1/ )
X
1
ˆ
)
ˆ
(
X 





Now try to solve for  as a function of , and call the
solution ̂
x
Method of Moments Estimator
Method of Moments : Solve for :

3. 3
Suppose there are two parameters to estimate (, ) and that
)
S
,
X
(
of
functions
are
)
ˆ
,
ˆ
(
o
S
)
ˆ
,
ˆ
h(
S
)
ˆ
,
ˆ
g(
X
that
so
ˆ
,
ˆ
find
s
MME'
2
2











E(X)= g(, ) Var(X)= h(, )
Key: solve 2 eqns
for 2 unknowns
Method of Moments Estimator

4. 4
MME: Easy Example
),
Poisson(
~
X
,...,
X
,
X n
2
1 
ˆ
X
)
ˆ
E(
then 

 


even I knew that!

5. 5
p)
,
Binomial(m
~
X
,...,
X
,
X n
2
1
S
-
1
X
m̂
)
S
-
m(1
X
S
-
1
p̂
p)
-
(1
X
S
:
Note
2
2
2
2
X
X
X






p)
-
mp(1
S
mp,
X 2


E(X)=mp, Var(X)=mp(1-p)
MME: Binomial Example
Both (m,p) are unknown

6. 6
0
x
,
e
x
)
(
f(x)
so
)
Gamma(r,
~
X
,...,
X -
1
-
r
n
1 

 x
r
r



MME: Gamma Example
2
2
ˆ
r̂
S
,
ˆ
r̂
X




2
r
Var(X)
,
r
E(X)





7. 7
MME: Gamma Example







̂
1
X
S2
Substitute!
2
2
2
S
X
X
ˆ
r̂
,
ˆ 

 

S
X

8. 8
If the probability density function for X is f(x; ), then if
sample items are obtained independently, the p.d.f. for
(X1,…,Xn) is (the joint density function)
f(x1,…xn; )= f(x1; )f(x2; )f(xn; )
Maximum Likelihood Estimation
We usually look at f as a function of (X1,…,Xn), but it is
also a function of the parameter . That is, we can
write L(; x1,…xn)= f(x1; )f(x2; )f(xn; )

9. 9
Maximum Likelihood Estimation
)
|
f(x
)
|
x
f( i
n
1
i
~

 


The MAXIMUM LIKELIHOOD ESTIMATOR (MLE)
of  is the value of  that maximizes L( | x )
L(; x1,…xn) is called the LIKELIHOOD of the data if we
write it as a function of parameters
)
|
f(x
)
x
|
L( i
1
~

 


n
i

10. 10
Hint : it is almost always easier to maximize the log of L rather
than L, and it maximizes at the same value of 
x
-
n
1 e
f(x)
so
)
Exp(
~
X
,...,
X 

 
Note : Check second derivative to make sure you didn’t find the
minimum likelihood estimator
MLE of  for Exponential
)
x
exp(-
e
)
x
|
L(
1
i
n
x
-
1
~
i






n
i
n
i



 

11. 11
MLE
the
is
X
1
ˆ
so
0
n
-
L
log
x
n
ˆ
at
0
x
-
n
L
log
x
-
)
nlog(
)
L(
log
2
2
2
n
1
i
i
n
1
i
i
n
1
i
i


























MLE of  for Exponential

12. 12
1) Identify f(x|) :
Computing the MLE: 6 Steps
)
x
(
f i
n
1
i



L() =
3) Write this as a function of the parameter L() :
2) Using data x1,…xn from the distribution in 1)
write the joint distribution f(x1,x2,…,xn| )

13. 13
4) To find the value of  that maximizes
L()= L(|x1,…xn),examine the log (ln L()) instead
Computing the MLE: 6 Steps
̂
6) Solve this equation for , call it . It is the MLE if the
second derivative is negative
0)
)
L(
ln
( 




5) Find max by taking derivative, set it to zero

14. 14
X1,X2,…,Xn~ Poisson()
 
















!
x
log
-
)log
x
(
n
-
!
log
log
x
-
L
log
!
e
)
(
)
L(
i
n
1
i
i
n
1
i
i
-
1
i
1
i







i
i
x
n
i
n
x
x
x
f
i
MLE for Poisson Distribution

15. 15
x
ˆ
at
0
n
-
L
log





 



i
x
MLE for Poisson Distribution
0.
x
-
L
log 2
i
2
2



 


Is this a maximum?
Oui!

16. 16




 


otherwise
0
x
0
1
f(x)
)
U(0,
~
X
,...,
X n
1



MLE for Uniform Distribution
)
x
I(0
)
x
I(0
1
)
f(x
)
x
,...,
x
,
x
|
L(
n
1
i
i
n
-
i
n
1
n
1
i
i
n
2
1


















i

17. 17
*
X
ˆ 

Same thing as I(0 < x* < ), where x* = Max(x1, x2 ,…, xn).
To make L() as Large as possible, choose  to be as small as
possible, but not so small that  < x* (otherwise L=0).
MLE for Uniform Distribution

18. 18











n
1
i
n
1
i
1)
(
-
i
n
n
1
i
i
log
)
1
(
-
nlog
)
L(
log
4.
)
L(
x
)
f(x
3.
&
2
i
x




 
To find the MLE of :
MLE Example: f(x) =  / x1
2 3 4 5
0.25
0.5
0.75
1
1.25
1.5
1.75
2
X1,…,Xn ~ f(x) =  / x1
where x > 1,  > 1

19. 19









n
1
i
n
1
i
ln
n
ˆ
log
-
n
)
L(
ln
5.
i
i
X
x 


MLE Example: f(x) =  / x1
So represents the MLE for 

ˆ
0
n
-
)
L(
ln
6. 2
2
2








20. 20
1
-
E(X)
1








dx
x
MME Example: f(x) =  / x1
1
-
X
1
ˆ
X
1
-
ˆ
ˆ


 


Note: in this
case, the MLE
and the MME
disagree.
MLE
MME