Upcoming SlideShare
×

On Approximating the Smallest Enclosing Bregman Balls

233 views

Published on

Slides for
On Approximating the Smallest Enclosing Bregman Balls

0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total views
233
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
1
0
Likes
0
Embeds 0
No embeds

No notes for slide

On Approximating the Smallest Enclosing Bregman Balls

1. 1. On Approximating the Smallest Enclosing Bregman Balls Frank Nielsen1 Richard Nock2 1 Sony Computer Science Laboratories, Inc. Fundamental Research Laboratory Frank.Nielsen@acm.org 2 University of Antilles-Guyanne DSI-GRIMAAG Richard.Nock@martinique.univ-ag.fr March 2006 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
2. 2. Smallest Enclosing Balls Problem Given S = {s1 , ..., sn }, compute a simpliﬁed description, called the center, that ﬁts well S (i.e., summarizes S). Two optimization criteria: M IN AVG Find a center c∗ which minimizes the average distortion w.r.t S: c∗ = argminc i d(c, si ). M IN M AX Find a center c∗ which minimizes the maximal distortion w.r.t S: c∗ = argminc maxi d(c, si ). Investigated in Applied Mathematics: Computational geometry (1-center problem), Computational statistics (1-point estimator), Machine learning (1-class classiﬁcation), F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
3. 3. Smallest Enclosing Balls in Computational Geometry Distortion measure d(·, ·) is the geometric distance: Euclidean distance L2 . c∗ is the circumcenter of S for M IN M AX, Squared Euclidean distance L2 . 2 c∗ is the centroid of S for M IN AVG (→ k -means), Euclidean distance L2 . c∗ is the Fermat-Weber point for M IN AVG. Centroid M IN AVG L2 2 Circumcenter M IN M AX L2 F. Nielsen and R. Nock Fermat-Weber M IN AVG L2 On Approximating the Smallest Enclosing Bregman Balls
4. 4. Core-sets for M IN M AX Ball ˘ → Introduced by Badoiu and Clarkson [BC’02] Approximating M IN M AX [BC’02]: ||c − c∗ || ≤ r ∗ A -approximation for the M IN M AX ball can be found in O( dn ) 2 time using algorithm BC. [for point/ball sets] Algorithm BC(S, T ) Input: S = {s1 , s2 , ..., sn } Output: Circumcenter c such that ||c − c∗ || ≤ Choose at random c ∈ S for t = 1, 2, ..., T do Find furthest point s ← arg maxs ∈S c − s Update circumcenter t 1 c ← t+1 c + t+1 s r∗ √ T 2 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
5. 5. Demo Algorithm BC: Initialization F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
6. 6. Demo Algorithm BC: Iteration 1 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
7. 7. Demo Algorithm BC: Iteration 2 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
8. 8. Demo Algorithm BC: Iteration 3 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
9. 9. Demo Algorithm BC: Iteration 4 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
10. 10. Demo Algorithm BC: Iteration 5 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
11. 11. Demo Algorithm BC: Iteration 6 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
12. 12. Demo Algorithm BC: Iteration 7 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
13. 13. Demo Algorithm BC: Iteration 8 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
14. 14. Demo Algorithm BC: Iteration 9 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
15. 15. Demo Algorithm BC: Summary After 10 iterations, we visualize the ball traces and the core-set. Ball traces Core-set Core-set’s size is independent of the dimension d. (depends only on 1 ) F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
16. 16. Distortions: Bregman Divergences Deﬁnition Bregman divergences are parameterized (F ) families of distortions. Let F : X −→ R, such that F is strictly convex and differentiable on int(X ), for a convex domain X ⊆ Rd . Bregman divergence DF : DF (x, y) = F (x) − F (y) − x − y, F ·, · : : F (y) . gradient operator of F Inner product (dot product) (→ DF is the tail of a Taylor expansion of F ) F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
17. 17. Visualizing F and DF F (·) DF (x, y) x − y, y F (y) x DF (x, y) = F (x) − F (y) − x − y, F (y) . (→ DF is the tail of a Taylor expansion of F ) F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
18. 18. Bregman Balls Two Bregman balls: Bc,r = {x ∈ X : DF ( c , x) ≤ r }, Bc,r = {x ∈ X : DF (x, c ) ≤ r } Euclidean Ball: Bc,r = {x ∈ X : x − c (r : squared radius. L2 : 2 2 2 ≤ r } = Bc,r Bregman divergence F (x) = d i=1 xi2 ) Lemma [NN’05] The smallest enclosing Bregman ball Bc∗ ,r ∗ of S is unique. Theorem [BMDG’04] The M IN AVG Ball for Bregman divergences is the centroid . F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
19. 19. Applying BC for divergences yields poor result −→ design a tailored algorithm for divergences. F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
20. 20. Bregman BC Algorithm BBC generalizes BC to Bregman divergences (analysis in [NN’05]). Algorithm BBC(S, T ) Choose at random c ∈ S for t = 1, 2, ..., T do Furthest point w.r.t. DF s ← arg maxs ∈S DF (c, s ) Circumcenter update t 1 c ← −1 t+1 F (c) + t+1 F F (s) Observations BBC(L2 ) is BC. 2 DF (c, x) is convex in c but not necessarily the ball’s boundary ∂Bc,r (depends on x given c; see Itakura-Saito ball). F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
21. 21. Demo BBC (Itakura-Saito): Initialization d DF (p, q) = ( i=1 pi p − log i − 1), [F (x) = − qi qi F. Nielsen and R. Nock d log xi ] i=1 On Approximating the Smallest Enclosing Bregman Balls
22. 22. Demo BBC (Itakura-Saito): Iteration 1 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
23. 23. Demo BBC (Itakura-Saito): Iteration 2 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
24. 24. Demo BBC (Itakura-Saito): Iteration 3 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
25. 25. Demo BBC (Itakura-Saito): Iteration 4 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
26. 26. Demo BBC (Itakura-Saito): Iteration 5 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
27. 27. Demo BBC (Itakura-Saito): Iteration 6 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
28. 28. Demo BBC (Itakura-Saito): Iteration 7 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
29. 29. Demo BBC (Itakura-Saito): Iteration 8 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
30. 30. Demo BBC (Itakura-Saito): Iteration 9 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
31. 31. Demo BBC(Itakura-Saito): Summary n = 100 points (d = 2) Sampling Ball All iterations F. Nielsen and R. Nock Core-set On Approximating the Smallest Enclosing Bregman Balls
32. 32. Demo BBC (Kullbach-Leibler): Initialization d DF (p, q) = i=1 p (pi log i − pi + qi ), [F (x) = − qi F. Nielsen and R. Nock d xi log xi ] i=1 On Approximating the Smallest Enclosing Bregman Balls
33. 33. Demo BBC (Kullbach-Leibler): Iteration 1 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
34. 34. Demo BBC (Kullbach-Leibler): Iteration 2 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
35. 35. Demo BBC (Kullbach-Leibler): Iteration 3 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
36. 36. Demo BBC (Kullbach-Leibler): Iteration 4 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
37. 37. Demo BBC (Kullbach-Leibler): Iteration 5 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
38. 38. Demo BBC (Kullbach-Leibler): Iteration 6 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
39. 39. Demo BBC (Kullbach-Leibler): Iteration 7 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
40. 40. Demo BBC (Kullbach-Leibler): Iteration 8 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
41. 41. Demo BBC (Kullbach-Leibler): Iteration 9 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
42. 42. Demo BBC(Kullbach-Leibler): Summary n = 100, d = 2. Sampling Ball All iterations F. Nielsen and R. Nock Core-set On Approximating the Smallest Enclosing Bregman Balls
43. 43. Fitting Bregman Balls For a same dataset (drawn from a 2D Gaussian distribution) Euclidean ball (L2 ), 2 Itakura-Saito ball, Kullbach-Leibler ball (as known as Information ball). Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
44. 44. BBC: Iteration 1 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
45. 45. BBC: Iteration 2 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
46. 46. BBC: Iteration 3 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
47. 47. BBC: Iteration 4 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
48. 48. BBC: Iteration 5 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
49. 49. BBC: Iteration 6 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
50. 50. BBC: Iteration 7 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
51. 51. BBC: Iteration 8 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
52. 52. BBC: Iteration 9 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
53. 53. BBC: After 10 iterations Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
54. 54. Experiments with BBC Kullbach-Leibler ball, 100 runs (n = 1000, T = 200 on the plane: d = 2) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 0 20 40 60 ∗ 80 100 120 140 160 180 200 ∗ Plain curves (BBC): DF (c ,c)+DF (c,c ) . 2 Dashed curves: Upperbound ||c − c∗ || from [BC’02]: F. Nielsen and R. Nock 1 T for L2 . 2 On Approximating the Smallest Enclosing Bregman Balls
55. 55. Bregman divergences ↔ Functional averages Bijection (core-sets) [NN’05] F (s) DF (c, s) cj (1 ≤ j ≤ d) Rd I Pd L2 norm 2 Pd 2 j=1 (cj − sj ) Arithmetic mean Pm i=1 αi si,j (I +,∗ )d R / d-simplex Pd Information/Kullbach-Leibler divergence Pd j=1 cj log(cj /sj ) − cj + sj Geometric mean Qm αi i=1 si,j (I +,∗ )d R − Rd I sT Cov−1 s p ∈ N {0, 1} I Itakura-Saito divergence Pd j=1 (cj /sj ) − log(cj /sj ) − 1 Mahalanobis divergence (c − s)T Cov−1 (c − s) Harmonic mean P 1/ m (αi /si,j ) i=1 Arithmetic mean Pm i=1 αi si,j Weighted power mean Domain R d /I + I R 2 j=1 sj d j=1 sj Pd j=1 (1/p) log sj − sj log sj Pd p j=1 sj Pd p c j j=1 p (p−1)s + p p j p−1 − cj sj Pm i=1 p−1 1/(p−1) αi si,j Bregman divergences ↔ Family of exponential distributions [BMDG’04]. F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
56. 56. References ˘ [BC’02] M. Badoiu and K. L. Clarkson, Core-sets for balls, Manuscript, 2002. (M IN M AX core-set and (1 + )-approximation) [NN’04] F. Nielsen and R. Nock, Approximating Smallest Enclosing Balls, ICCSA 2004. (Survey on M IN M AX) [BMDG’04] A. Banerjee, S. Merugu, I. S. Dhillon, J. Ghosh, Clustering with Bregman Divergences, SDM 2004. (Bregman divergences and k-means) [NN’05] R. Nock and F. Nielsen, Fitting the Smallest Enclosing Bregman Ball, ECML 2005. (Bregman Balls) F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
57. 57. Linux R and Windows R codes Available from our Web pages: http://www.csl.sony.co.jp/person/nielsen/ BregmanBall/ http://www.univ-ag.fr/∼rnock/BregmanBall/ Command line executable: cluster -h -B display this help choose Bregman divergence (0: Itakura-Saito, 1: Euclidean, 2: Kullbach-Leibler (simplex), 3: Kullbach-Leibler (general), 4: Csiszar) -d -n -k -g dimension number of points number of theoretical clusters type of clusters (0: Gaussian, 1: Ring Gaussian, 2: Uniform, 3: Gaussian with small σ, 4: Uniform Bregman) -S ... number of runs ... F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls