Paper presentation

1. Dependency Parsing by Belief
Propagation.
Presenter: Roy Adams
David A. Smith and Jason Eisner

2. Undirected Graphical Models
Y1 Y2 Y3 Y4
X1 X2 X3 X4
= Variable not observed at test time.
= Variable always observed.
= Factor. Encodes correlations between variables.

3. Undirected Graphical Models
Y1 Y2 Y3 Y4
X1 X2 X3 X4
= Variable not observed at test time.
= Variable always observed.

4. Undirected Graphical Models
Y1 Y2 Y3 Y4
X1 X2 X3 X4

5. Undirected Graphical Models
Y1 Y2 Y3 Y4
X1 X2 X3 X4

6. Undirected Graphical Models
Y1 Y2 Y3 Y4
X1 X2 X3 X4

7. Marginal Inference
Y1 Y2 Y3 Y4
X1 X2 X3 X4

8. Sum-Product Message Passing
Initialize all messages to 1
Until Convergence:
For each variable, Yi:
For each F in Neighborhood(Yi):
For each v in Dom(Yi):
For each factor, F:
For each Yi in Neighborhood(F):
For each v in Dom(Yi):

9. Sum-Product Message Passing
Initialize all messages to 1
Until Convergence:
For each variable, Yi:
For each F in Neighborhood(Yi):
For each v in Dom(Yi):
For each factor, F:
For each Yi in Neighborhood(F):
For each v in Dom(Yi):

10. Sum-Product Message Passing
Initialize all messages to 1
Until Convergence:
For each variable, Yi:
For each F in Neighborhood(Yi):
For each v in Dom(Yi):
For each factor, F:
For each Yi in Neighborhood(F):
For each v in Dom(Yi):

11. Sum-Product Message Passing
Initialize all messages to 1
Until Convergence:
For each variable, Yi:
For each F in Neighborhood(Yi):
For each v in Dom(Yi):
For each factor, F:
For each Yi in Neighborhood(F):
For each v in Dom(Yi):

12. Sum-Product Message Passing
Yi
“Based on what my other neighbors said,
this my distribution over my values.”

13. Sum-Product Message Passing
Yi
“Based on what my other neighbors said,
this my distribution over your values.”

14. Sum-Product Message Passing
- If it converges, we get:
- For tree structured graphs, it is guaranteed to converge
and the marginals will be exact.
- For loopy graphs, it may not converge, and if it
converges, the marginals may not be correct.
- With some tricks, it tends to work very well in
practice.

15. Questions so far?

16. High Order Factors
Y1 Y3 YnY2 …

17. Why do we want them?
Y1 Y3 YnY2 …
They can be used to encode structure. E.g. Y1
through Yn must all take different values.

18. Why are they hard?
Y1 Y3 YnY2 …

19. Why are they hard?
Y1 Y3 YnY2 …
Basic message passing has exponential
complexity in the neighborhood size of the
largest factor.

20. Fixed High Order Factors
Y1 Y3 YnY2 …

21. Fixed High Order Factors
1) Doesn’t depend on the parameters, so we don’t
need the marginal to calculate gradients in MLE.

22. Fixed High Order Factors
2) The structure is so constrained, that the sum in
message passing becomes tractable.

23. Other Structures (Smith and Eisner)
-Parse trees
-Unique labels
-Label ordering
-Segmentation

24. Questions?