Your SlideShare is downloading. ×
On Approximating the Smallest Enclosing Bregman Balls
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

On Approximating the Smallest Enclosing Bregman Balls

95
views

Published on

Slides for …

Slides for
On Approximating the Smallest Enclosing Bregman Balls
See also http://www.youtube.com/watch?v=ROdWANRAYHQ

Published in: Technology, Business

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
95
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. On Approximating the Smallest Enclosing Bregman Balls Frank Nielsen1 Richard Nock2 1 Sony Computer Science Laboratories, Inc. Fundamental Research Laboratory Frank.Nielsen@acm.org 2 University of Antilles-Guyanne DSI-GRIMAAG Richard.Nock@martinique.univ-ag.fr March 2006 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 2. Smallest Enclosing Balls Problem Given S = {s1 , ..., sn }, compute a simplified description, called the center, that fits well S (i.e., summarizes S). Two optimization criteria: M IN AVG Find a center c∗ which minimizes the average distortion w.r.t S: c∗ = argminc i d(c, si ). M IN M AX Find a center c∗ which minimizes the maximal distortion w.r.t S: c∗ = argminc maxi d(c, si ). Investigated in Applied Mathematics: Computational geometry (1-center problem), Computational statistics (1-point estimator), Machine learning (1-class classification), F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 3. Smallest Enclosing Balls in Computational Geometry Distortion measure d(·, ·) is the geometric distance: Euclidean distance L2 . c∗ is the circumcenter of S for M IN M AX, Squared Euclidean distance L2 . 2 c∗ is the centroid of S for M IN AVG (→ k -means), Euclidean distance L2 . c∗ is the Fermat-Weber point for M IN AVG. Centroid M IN AVG L2 2 Circumcenter M IN M AX L2 F. Nielsen and R. Nock Fermat-Weber M IN AVG L2 On Approximating the Smallest Enclosing Bregman Balls
  • 4. Core-sets for M IN M AX Ball ˘ → Introduced by Badoiu and Clarkson [BC’02] Approximating M IN M AX [BC’02]: ||c − c∗ || ≤ r ∗ A -approximation for the M IN M AX ball can be found in O( dn ) 2 time using algorithm BC. [for point/ball sets] Algorithm BC(S, T ) Input: S = {s1 , s2 , ..., sn } Output: Circumcenter c such that ||c − c∗ || ≤ Choose at random c ∈ S for t = 1, 2, ..., T do Find furthest point s ← arg maxs ∈S c − s Update circumcenter t 1 c ← t+1 c + t+1 s r∗ √ T 2 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 5. Demo Algorithm BC: Initialization F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 6. Demo Algorithm BC: Iteration 1 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 7. Demo Algorithm BC: Iteration 2 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 8. Demo Algorithm BC: Iteration 3 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 9. Demo Algorithm BC: Iteration 4 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 10. Demo Algorithm BC: Iteration 5 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 11. Demo Algorithm BC: Iteration 6 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 12. Demo Algorithm BC: Iteration 7 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 13. Demo Algorithm BC: Iteration 8 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 14. Demo Algorithm BC: Iteration 9 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 15. Demo Algorithm BC: Summary After 10 iterations, we visualize the ball traces and the core-set. Ball traces Core-set Core-set’s size is independent of the dimension d. (depends only on 1 ) F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 16. Distortions: Bregman Divergences Definition Bregman divergences are parameterized (F ) families of distortions. Let F : X −→ R, such that F is strictly convex and differentiable on int(X ), for a convex domain X ⊆ Rd . Bregman divergence DF : DF (x, y) = F (x) − F (y) − x − y, F ·, · : : F (y) . gradient operator of F Inner product (dot product) (→ DF is the tail of a Taylor expansion of F ) F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 17. Visualizing F and DF F (·) DF (x, y) x − y, y F (y) x DF (x, y) = F (x) − F (y) − x − y, F (y) . (→ DF is the tail of a Taylor expansion of F ) F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 18. Bregman Balls Two Bregman balls: Bc,r = {x ∈ X : DF ( c , x) ≤ r }, Bc,r = {x ∈ X : DF (x, c ) ≤ r } Euclidean Ball: Bc,r = {x ∈ X : x − c (r : squared radius. L2 : 2 2 2 ≤ r } = Bc,r Bregman divergence F (x) = d i=1 xi2 ) Lemma [NN’05] The smallest enclosing Bregman ball Bc∗ ,r ∗ of S is unique. Theorem [BMDG’04] The M IN AVG Ball for Bregman divergences is the centroid . F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 19. Applying BC for divergences yields poor result −→ design a tailored algorithm for divergences. F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 20. Bregman BC Algorithm BBC generalizes BC to Bregman divergences (analysis in [NN’05]). Algorithm BBC(S, T ) Choose at random c ∈ S for t = 1, 2, ..., T do Furthest point w.r.t. DF s ← arg maxs ∈S DF (c, s ) Circumcenter update t 1 c ← −1 t+1 F (c) + t+1 F F (s) Observations BBC(L2 ) is BC. 2 DF (c, x) is convex in c but not necessarily the ball’s boundary ∂Bc,r (depends on x given c; see Itakura-Saito ball). F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 21. Demo BBC (Itakura-Saito): Initialization d DF (p, q) = ( i=1 pi p − log i − 1), [F (x) = − qi qi F. Nielsen and R. Nock d log xi ] i=1 On Approximating the Smallest Enclosing Bregman Balls
  • 22. Demo BBC (Itakura-Saito): Iteration 1 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 23. Demo BBC (Itakura-Saito): Iteration 2 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 24. Demo BBC (Itakura-Saito): Iteration 3 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 25. Demo BBC (Itakura-Saito): Iteration 4 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 26. Demo BBC (Itakura-Saito): Iteration 5 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 27. Demo BBC (Itakura-Saito): Iteration 6 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 28. Demo BBC (Itakura-Saito): Iteration 7 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 29. Demo BBC (Itakura-Saito): Iteration 8 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 30. Demo BBC (Itakura-Saito): Iteration 9 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 31. Demo BBC(Itakura-Saito): Summary n = 100 points (d = 2) Sampling Ball All iterations F. Nielsen and R. Nock Core-set On Approximating the Smallest Enclosing Bregman Balls
  • 32. Demo BBC (Kullbach-Leibler): Initialization d DF (p, q) = i=1 p (pi log i − pi + qi ), [F (x) = − qi F. Nielsen and R. Nock d xi log xi ] i=1 On Approximating the Smallest Enclosing Bregman Balls
  • 33. Demo BBC (Kullbach-Leibler): Iteration 1 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 34. Demo BBC (Kullbach-Leibler): Iteration 2 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 35. Demo BBC (Kullbach-Leibler): Iteration 3 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 36. Demo BBC (Kullbach-Leibler): Iteration 4 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 37. Demo BBC (Kullbach-Leibler): Iteration 5 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 38. Demo BBC (Kullbach-Leibler): Iteration 6 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 39. Demo BBC (Kullbach-Leibler): Iteration 7 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 40. Demo BBC (Kullbach-Leibler): Iteration 8 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 41. Demo BBC (Kullbach-Leibler): Iteration 9 F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 42. Demo BBC(Kullbach-Leibler): Summary n = 100, d = 2. Sampling Ball All iterations F. Nielsen and R. Nock Core-set On Approximating the Smallest Enclosing Bregman Balls
  • 43. Fitting Bregman Balls For a same dataset (drawn from a 2D Gaussian distribution) Euclidean ball (L2 ), 2 Itakura-Saito ball, Kullbach-Leibler ball (as known as Information ball). Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 44. BBC: Iteration 1 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 45. BBC: Iteration 2 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 46. BBC: Iteration 3 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 47. BBC: Iteration 4 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 48. BBC: Iteration 5 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 49. BBC: Iteration 6 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 50. BBC: Iteration 7 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 51. BBC: Iteration 8 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 52. BBC: Iteration 9 Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 53. BBC: After 10 iterations Squared Euclidean Itakura-Saito F. Nielsen and R. Nock Kullbach-Leibler On Approximating the Smallest Enclosing Bregman Balls
  • 54. Experiments with BBC Kullbach-Leibler ball, 100 runs (n = 1000, T = 200 on the plane: d = 2) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 0 20 40 60 ∗ 80 100 120 140 160 180 200 ∗ Plain curves (BBC): DF (c ,c)+DF (c,c ) . 2 Dashed curves: Upperbound ||c − c∗ || from [BC’02]: F. Nielsen and R. Nock 1 T for L2 . 2 On Approximating the Smallest Enclosing Bregman Balls
  • 55. Bregman divergences ↔ Functional averages Bijection (core-sets) [NN’05] F (s) DF (c, s) cj (1 ≤ j ≤ d) Rd I Pd L2 norm 2 Pd 2 j=1 (cj − sj ) Arithmetic mean Pm i=1 αi si,j (I +,∗ )d R / d-simplex Pd Information/Kullbach-Leibler divergence Pd j=1 cj log(cj /sj ) − cj + sj Geometric mean Qm αi i=1 si,j (I +,∗ )d R − Rd I sT Cov−1 s p ∈ N {0, 1} I Itakura-Saito divergence Pd j=1 (cj /sj ) − log(cj /sj ) − 1 Mahalanobis divergence (c − s)T Cov−1 (c − s) Harmonic mean P 1/ m (αi /si,j ) i=1 Arithmetic mean Pm i=1 αi si,j Weighted power mean Domain R d /I + I R 2 j=1 sj d j=1 sj Pd j=1 (1/p) log sj − sj log sj Pd p j=1 sj Pd p c j j=1 p (p−1)s + p p j p−1 − cj sj  Pm i=1  p−1 1/(p−1) αi si,j Bregman divergences ↔ Family of exponential distributions [BMDG’04]. F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 56. References ˘ [BC’02] M. Badoiu and K. L. Clarkson, Core-sets for balls, Manuscript, 2002. (M IN M AX core-set and (1 + )-approximation) [NN’04] F. Nielsen and R. Nock, Approximating Smallest Enclosing Balls, ICCSA 2004. (Survey on M IN M AX) [BMDG’04] A. Banerjee, S. Merugu, I. S. Dhillon, J. Ghosh, Clustering with Bregman Divergences, SDM 2004. (Bregman divergences and k-means) [NN’05] R. Nock and F. Nielsen, Fitting the Smallest Enclosing Bregman Ball, ECML 2005. (Bregman Balls) F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls
  • 57. Linux R and Windows R codes Available from our Web pages: http://www.csl.sony.co.jp/person/nielsen/ BregmanBall/ http://www.univ-ag.fr/∼rnock/BregmanBall/ Command line executable: cluster -h -B display this help choose Bregman divergence (0: Itakura-Saito, 1: Euclidean, 2: Kullbach-Leibler (simplex), 3: Kullbach-Leibler (general), 4: Csiszar) -d -n -k -g dimension number of points number of theoretical clusters type of clusters (0: Gaussian, 1: Ring Gaussian, 2: Uniform, 3: Gaussian with small σ, 4: Uniform Bregman) -S ... number of runs ... F. Nielsen and R. Nock On Approximating the Smallest Enclosing Bregman Balls