The Ninth Workshop on Information Theoretic Methods in Science and Engineering (WITMSE), Helsinki, Finland, on September 19–21, 2016.

1. Jeffreys' and BDeu Priors for Model Selection
WITMSE 2016
Helsinki, Finland, September 20
Joe Suzuki
(prof-joe)
Joe Suzuki (Osaka Univ., Japan)

2. Goal and Contributions
[Goal]
Compare for model selection
• BDeu (Bayesian Dirichlet equivalent uniform)
• Jeffreys prior (T-K estimator)
[Contribution]
Mathematically Proves

3. Road Map
1. Bayesian Dirichlet Scores
2. BDeu and Jeffreys Scores
3. A Found Property and its Proof
4. Main Theorem
5. Regularity in Model Selection
6. Summary

4. Assign a Prob. to each Seq.

5. Express a Prob. by the product of Cond. Probs.

6. Simultaneous Probs.

7. Cond. Probs.

8. BDeu and Jeffreys’ Prior

10. Example 1 : Bayesian Network Structure Learning (BNSL)

11. Example 2: Independence Testing

12. A Motivating Example

13. A Found Property

14. Sketch of J(n)>0 for BDeu

15. Sketch of J(n)≦0 for Jeffreys’

16. An Intuitive Reasoning

17. Main Theorem

18. Examples
more likely
unlikely

19. Regularity in Model Selection
Fitness + Simplicity → optimal
(-1) x Likelihood + Penalty Term → min
Newton’s
Law of
Motion
Maxwell
Equations
If model A is better than model B w.r.t. fitness and simplicity,
model A should be chosen (regularity).
Information Criteria
LASSO

20. BDeu violates regularity in model selection
Z XZ X
Y
Y X

21. B&B for efficient BNSL (Depth First Search)

22. Those bounds utilize regularity
Campos and Ji 2011 figured out one (=nice)
but the bound is not efficient (experiments).
Designing Pruning rules for BDeu is HARDer.
because regularity cannot be assumed

23. Bayes Prior
Based on his/her Belief:
Nobody should reject it from a general point of view.
BDeu violates regularity
contradicts with Newton, Maxwell, Information Critreria, LASSO, etc.
People might notice that their beliefs have been
wrong, after knowing the new result in this paper.

24. Summary
The prior behind BDeu might have been based on a wrong belief
That contradicts regularity in model selection
Future Work: Consider NML and others in a similar way