This document discusses estimation theory and the maximum likelihood principle. It introduces key concepts such as: - Estimating unknown parameters from a set of data to obtain the highest probability of the observed data. - The maximum likelihood principle estimates parameters such that the probability of obtaining the actual observed sample is maximized. - The Fisher information and Cramer-Rao lower bound place a theoretical lower bound on the variance of unbiased estimators, with the maximum likelihood estimate achieving this lower bound.

1. Estimation Theory1

2. Estimation TheoryWe seek to determine from a set of data, a set of parameters such that their values would yield the highest probability of obtaining the observed data.The unknown parameters may be seen as deterministic or random variablesThere are essentially two alternatives to the statistical caseWhen no a priori distribution assumed then Maximum LikelihoodWhen a priori distribution known then Bayes

3. Maximum LikelihoodPrinciple: Estimate a parameter such that for this value the probability of obtaining an actually observed sample is as large as possible.I.e. having got the observation we “look back” and compute probability that the given sample will be observed, as if the experiment is to be done again.This probability depends on a parameter which is adjusted to give it a maximum possible value.Reminds you of politicians observing the movement of the crowd and then move to the front to lead them?

4. Estimation TheoryLet a random variable have a probability distribution dependent on a parameter The parameter lies in a space of all possible parameters Let be the probability density function of Assume the the mathematical form of is known but not

5. Estimation TheoryThe joint pdf of sample random variables evaluated at each the sample pointsIs given asThe above is known as the likelihood of the sampled observation

6. Estimation TheoryThe likelihood function is a function of the unknown parameter for a fixed set of observationsThe Maximum Likelihood Principle requires us to select that value of that maximises the likelihood functionThe parameter may also be regarded as a vector of parameters

7. Estimation TheoryIt is often more convenient to useThe maximum is then at

8. An exampleLet be a random sample selected from a normal distributionThe joint pdf is We wish to find the best and

9. Estimation TheoryForm the log-likelihood functionHenceor

10. Fisher and Cramer-RaoThe Fisher Information helps in placing a bound on estimatorsCramer-Rao Lower Bound:“If is any unbiased estimator of based on maximum likelihood then Ie provides a lower bound on the covariance matrix of any unbiased estimator

11. Estimation TheoryIt can be seen that if we model the observations as the output of an AR process driven by zero mean Gaussian noise then the Maximum Likelihood estimator for the variance is also the Least Squares Estimator.

12. The Cramer-Rao Lower BoundThis is an important theorem which establishes the superiority of the ML estimate over all others. The Cramer-Rao lower bound is the smallest theoretical variance which can be achieved. ML gives this so any other estimation technique can at best only equal it. this is the Cramer-Rao inequality.

13. CRB Definition:

14. Inverse of the Fisher Matrix:

15. lowest possible variance

16. Purpose of CRB analysis:

17. indicate the performance bounds of a particular problem.

18. facilitate analysis of factors that impact most on the performance of an algorithm.

19. Fisher Matrix of our Energy Decay ModelThe Cramer-Rao Lower Bound

20. A different way of looking at information for continuous functions r(s).

21. Intuition: We can discriminate values of s better if r(s) is changing rapidly. If r(s) does not change much with s, then we don’t learn much about s.The Cramer-Rao Lower Bound

22. How much does r tell us about s?

23. Fisher information is high when:

24. High negative curvature at s for all r (on average)

25. Rapid change in p(r|s) at s for all r (on average)

26. Implies easy to discriminate different values of sThe Cramer-Rao Lower Bound

27. Applies to unbiased estimators for which

28. The best unbiased estimator is only so good.

29. The best unbiased estimator is the ML estimate

30. Show a relation between variance of estimators and an information measureThe Cramer-Rao Lower Bound