1. Maximum Entropy Model LING 572 Fei Xia 02/07-02/09/06

5. Modeling

8. Features in POS tagging (Ratnaparkhi, 1996) context (a.k.a. history) allowable classes

11. Coin-flip example (cont) p1 p2 H p1 + p2 = 1 p1+p2=1.0, p1=0.3

12. Ex2: An MT example (Berger et. al., 1996) Possible translation for the word “in” is: Constraint: Intuitive answer:

13. An MT example (cont) Constraints: Intuitive answer:

14. An MT example (cont) Constraints: Intuitive answer: ??

15. Ex3: POS tagging (Klein and Manning, 2003)

16. Ex3 (cont)

17. Ex4: overlapping features (Klein and Manning, 2003)

20. Some notations Finite training sample of events: Observed probability of x in S: The model p’s probability of x: Model expectation of : Observed expectation of : The j th feature: (empirical count of )

22. Training data  observed events

23. Restating the problem The task: find p* s.t. where Objective function: -H(p) Constraints: Add a feature

25. What is the form of p*? (Ratnaparkhi, 1997) Theorem: if then Furthermore, p* is unique.

26. Using Lagrangian multipliers Minimize A(p):

27. Two equivalent forms

28. Relation to Maximum Likelihood The log-likelihood of the empirical distribution as predicted by a model q is defined as Theorem: if then Furthermore, p* is unique.

29. Summary (so far) The model p* in P with maximum entropy is the model in Q that maximizes the likelihood of the training sample Goal: find p* in P, which maximizes H(p). It can be proved that when p* exists it is unique.

31. Parameter estimation

35. Approximation for calculating feature expectation

38. Calculating If Then GIS is the same as IIS Else must be calcuated numerically.

39. Feature selection

41. Notation The gain in the log-likelihood of the training data: After adding a feature: With the feature set S:

45. Case study

48. Modeling

49. Training step 1: define feature templates History h i Tag t i

53. Beam search

54. Viterbi search

56. Experiment results

59. Additional slides

60. Ex4 (cont) ??