Voronoi diagrams in information geometry: Statistical Voronoi diagrams and their applications

Voronoi diagrams in information geometry
Statisti
al Voronoi diagrams and their appli
ations
Frank NIELSEN
É
ole Polyte
hnique
Sony CSL
e-mail: Frank.Nielsen a
m.org
MaxEnt 2014

2014 Frank NIELSEN 5793b870 1/57

Outline of the tutorial
1. Eu
lidean Voronoi diagrams
2. Dis
riminant analysis, Mahalanobis Voronoi diagrams, and
anisotropi
Voronoi diagrams
3. Fisher-Hotelling-Rao Voronoi diagrams (Riemannian
urved geometries)
4. Kullba
k-Leibler Voronoi diagrams (Bregman Voronoi
diagrams, dually at geometries)
5. Bayes' error, Cherno information and statisti
al Voronoi
diagrams


Eu
lidean (ordinary) Voronoi diagrams
P = {P1, ..., Pn} : n distin
t point generators in Eu
lidean spa
e Ed
=1Bi+(Pi , Pj )
V (Pi ) = {X : DE (Pi ,X) ≤ DE (Pj ,X), ∀j6= i} = ∩ni
DE (P,Q) = k(P) − (Q)k2 =
qPd
i=1(i (P) − i (Q))2
(P) = p : Cartesian
oordinate system with j (Pi ) = p(j)
i .
Bise
tors Bi(P,Q) = {X : DE (P,X) ≤ DE (Q,X)} : hyperplanes
Voronoi diagram =
ell
omplex V (Pi )'s with their fa
es
⇒ Many appli
ations :
rystal growth,
odebook/quantization,
mole
ule interfa
es/do
king, motion planning, et
.


Voronoi diagrams and dual Delaunay simpli
ial
omplex
⇒ Empty sphere property, max min angle triangulation, et
⇒ General position : no (d + 2) points
ospheri
al
⇒ Voronoi dual Delaunay triangulation
⇒ Bise
tor Bi(P,Q) perpendi
ular ⊥ to segment [PQ]


Minimum spanning tree ⊂ Delaunay triangulation
All edges of the Eu
lidean MST are Delaunay edges
→ Prim's greedy algorithm in Delaunay graph in O(n log n)
Convex hull : boundary @ of the Delaunay triangulation


Order k Voronoi diagrams
All subsets of size k, Pk =
P
k

= {K1, ...,KN} with N =
n
k

.
partition the spa
e into non-empty k-order Voronoi
ells :
Vork (Ki ) = {x : ∀q ∈ Ki , ∀r ∈ PKi , D(x, q) ≤ D(x, r )}
Combinatorial
omplexity not yet settled ! ! ! (k-sets/levels)
≡ proje
tion of k-levels of arrangements of hyperplanes in Rd+1.


Farthest order Voronoi and Minimum En
losing Ball
Cir
um
enter/minmax
enter on the farthest Voronoi diagram.
Non-dierentiable optim. at the farthest Voronoi [16℄


Voronoi Delaunay : Complexity and algorithms
◮ Complexity : (n⌈ d
2 ⌉) (quadrati
in 3D)
Moment
urve : t7→ (t, t2, .., td ), et
.
◮ Constru
tion : (n log n + n⌈ d
2 ⌉), output-sensitive algorithms

(n log n + f ), not yet optimal output-sensitive
algorithms.
◮ Voronoi diagram : VorD(P) = Vorf (D(·,·))(P) for a stri
tly
monotoni
ally in
reasing fun
tion f (·)
Vor. Eu
lidean ≡ Vor. squared Eu
lidean ⊂ Vor. Bregman
◮ Geometri
toolbox : spa
e of spheres (polarity, orthogonality
between spheres, et
.)


Multiple Hypothesis Testing [9, 10℄
Given a random variable X with n hypothesis
H1 : X ∼ P1
: ...
Hn : X ∼ Pn
de
ide for a IID sample x1, ..., xm ∼ X whi
h hypothesis holds true ?
Pm
orre
t = 1 − Pm
error
Asymptoti
regime : Error exponent
lim
m→∞
−
1
m
log Pm
e =


Bayesian hypothesis testing
◮ Prior probabilities : wi = P(X ∼ Pi ) 0 (with
Pn
i=1 wi = 1)
◮ Conditional probabilities : P(X = x|X ∼ Pi ).
P(X = x) =
Xn
i=1
P(X ∼ Pi )P(X = x|X ∼ Pi ) =
Xn
i=1
wiP(X|Pi )
◮ Cost design matrix [cij ]= with ci ,j =
ost of de
iding Hi when
in fa
t Hj is true
◮ pi ,j (u(x)) = probability of making this de
ision using rule u.

2014 Frank NIELSEN 5793b870 10/57

Bayesian dete
tor : MAP rule
◮ Minimize the expe
ted
ost :
EX [c(r (x))], c(r (x)) =
X
i

wi
X
j6=i


ci ,jpi ,j (r (x))
◮ Spe
ial
ase : Probability of error Pe : ci ,i = 0 and ci ,j = 1 for
i6= j :
Pe = EX


X
i

wi
X
j6=i


pi ,j (r (x))


◮ Maximum a posteriori probability (MAP) rule :
map(x) = argmaxi∈{1,...,n} wipi (x)
where pi (x) = P(X = x|X ∼ Pi ) are the
onditional
probabilities.
→ MAP Bayesian dete
tor minimizes Pe over all rules [7℄

2014 Frank NIELSEN 5793b870 11/57

MVNs, MAP rule, and Mahalanobis distan
e (1930)
MultiVariate Normals
(MVNs)
◮ Say, X1 ∼ N(μ1,) and X2 ∼ N(μ2,)
◮ Probability of error (mis
lassi
ation) for w1 = w2 = 12
:
Pe =
1
2
(1 − TV(P1, P2)) =

−
1
2

(P1, P2)
(·) : standard normal distribution fun
tion
◮ Mahalanobis metri
distan
e (·, ·) :
2(X1,X2) = (μ1 − μ2)⊤−1(μ1 − μ2) = μ⊤−1μ
◮ Measure the density overlapping (the greater, the lesser)
◮ generalize Eu
lidean distan
e ( = I )
◮ 2 : only symmetri
Bregman divergen
e.
⇒ easy to approximate P˜e experimentally

2014 Frank NIELSEN 5793b870 12/57

Mahalanobis Voronoi diagrams
MHT for isotropi
MVNs with uniform weight wi =
1
n
Cholesky de
omposition = LL⊤
2(X1,X2) = (μ1 − μ2)⊤−1(μ1 − μ2),
= (μ1 − μ2)⊤(L−1)⊤L−1(μ1 − μ1),
= (L−1μ1 − L−1μ2)⊤(L−1μ1 − L−1μ2)
2(X1,X2) = DE (L−1μ1, L−1μ2)
Mahalanobis Voronoi diagram :
◮ Cholesky de
omposition : = LL⊤
◮ Map P = {p1, ..., pn} to P′ = {p′ 1
, ..., p′n
i = L−1x.
} with p′
◮ Ordinary Voronoi diagram VorE (P′)
◮ Map VorE (P′) to Vor(P) = LVorE (P′).

2014 Frank NIELSEN 5793b870 13/57

Mahalanobis Voronoi diagrams
a
ount for both
orrelation and dimension (feature) s
aling
Dual stru
ture ≡ anisotropi
Delaunay triangulation
⇒ empty
ir
umellipse property
Re
all : MAP rule from Voronoi diagram

2014 Frank NIELSEN 5793b870 14/57

Anisotropi
Voronoi diagrams [6℄ (MHT for MVNs)
Classi
ation : Generator Xi ∼ N(μi ,i ), wi = 1
n .
Dis
riminant fun
tions are quadrati
bise
tors
Orphans, islands, dual not a triangulation
(need wedged/visibility
onditions)
Appli
ations : Crystal growth in elds (not Riemannian smooth
metri
)

2014 Frank NIELSEN 5793b870 15/57

Statisti
s : Estimators
Given x1, ..., xn IID observations (from a population), estimate the
underlying distribution X ∼ p.
◮ empiri
al CDF : ˆ Fn(x) = 1
n
P
i I(−∞,x)(xi ).
Glivenko-Cantelli theorem : supx | ˆ Fn(x) − F(x)| → 0 a.s.
◮ Fisher approa
h : Density p belongs to a parametri
family
p(x|), D = #parameters (order)
◮ Method of moments : Mat
h distribution moments with
sample moments :
EX [Xl ] =
1
n
Xn
i=1
xl
i ,
⇒ any D independent equations yields an estimator
◮ Fisher Maximum (log-)Likelihood Estimator MLE :
max

l (; x1, ..., xn) = max

Yn
i=1
p(xi ; )

2014 Frank NIELSEN 5793b870 16/57

Estimation : Varian
e Lower Bound
Whi
h estimator ˆ n shall we
hoose ?
Estimator is also a random variable on a random ve
tor
◮ Consistent : limn→∞ ˆn →
◮ Unbiased : E[ˆn] − = 0
◮ Minimum square error (MSE) : E[(ˆ − )2] = B[ˆ]2 + V[ˆ]
◮ Fisher information matrix :
I () =

Ii ,j () = E

@
@i
log p(x|)
@
@j

log p(x|)
◮ Fré
het Darmois Cramér-Rao lower bound for unbiased
estimators :
V[ ˆ n]
1
n
I−1()
◮ E
ient : estimator rea
hes the Cramèr-Rao lower bound

2014 Frank NIELSEN 5793b870 17/57

Su
ient statisti
s and exponential families
◮ Su
ient statisti
t(x) :
P(x|t(x) = t, ) = P(x|t(x) = t)
All information for inferen
e
ontained in t(x). (6= an
illary)
◮ For univariate Gaussians, D = 2 : t1(x) = x and t2(x) = x2.
ˆμ = ¯x =
1
n
X
i
xi , ˆ2 =
1
n
X
i
(xi −¯x)2 =

1
n
X
i
x2
i
!
−(¯x)2
◮ Exponential families have nite dimensional su
ient
statisti
s (Koopman) : Redu
e n data to D statisti
s.
∀x ∈ X, P(x|) = exp(⊤t(x) − F() + k(x))
F(·) : log-normalizer/
umulant/partition fun
tion
k(x) : auxiliary term for
arrier measure

2014 Frank NIELSEN 5793b870 18/57

Exponential families : Fisher information and MLE
Common distributions are exponential families :
Poisson, Gaussians, Gamma, Beta, Diri
hlet, et
.
◮ Fisher information :
I () = ∇2F()
◮ MLE for an exponential family :
∇F() =
1
n
X
i
t(Xi ) =
MLE exists i.
= ∇F() ∈ int(CH(X))

2014 Frank NIELSEN 5793b870 19/57

Population spa
e : Hotelling (1930) [5℄ Rao (1945) [25℄
Birth of dierential-geometri
methods in statisti
s.
◮ Fisher information matrix (positive denite)
an be used as a
(smooth) Riemannian metri
tensor.
◮ Distan
e between two populations indexed by 1 and 2 :
Riemannian distan
e (metri
length)
◮ Fisher-Hotelling-Rao (FHR) geodesi
distan
e used in
lassi
ation : Find the
losest population to a given
population
◮ Used in tests of signi
an
e (null versus alternative
hypothesis), power of a test : P(reje
t H0|H0 is false )
(surfa
es in population spa
es)

2014 Frank NIELSEN 5793b870 20/57

Rao's distan
e (1945, introdu
ed by Hotelling 1930 [5℄)
◮ Innitesimal length element :
ds2 =
X
ij
gij ()didj = dT I ()d
◮ Geodesi
and distan
e are hard to expli
itly
al
ulate :
(p(x; 1), p(x; 2)) = min
(s)
(0)=1
(1)=2
Z 1
0
s
d
ds
T
I ()
d
ds
ds
◮ Metri
property of , many tools [1℄ : Riemannian Log/Exp
tangent/manifold mapping

2014 Frank NIELSEN 5793b870 21/57

Fisher Rao Hotelling Voronoi : Riemannian Voronoi diagrams
◮ Lo
ation-s
ale 2D families have non-positive
urvature
(Hotelling, 1930) : FHR Voronoi diagrams amount to
hyperboli
Voronoi diagrams or Eu
lidean diagrams
(lo
ation families only like isotropi
Gaussians)
◮ Multinomial family has spheri
al geometry on the positive
orthant : Spheri
al Voronoi diagram (stereographi
proje
tion ∝ Eu
lidean Voronoi diagrams)
◮ Arbitrary families p(x|) : Geodesi
s not in
losed forms →
limited
omputational framework in pra
ti
e...

2014 Frank NIELSEN 5793b870 22/57

Hyperboli
Voronoi diagrams [19, 22℄
◮ In Klein disk, the hyperboli
Voronoi diagram amounts to a
lipped ane Voronoi diagram, or a
lipped power
diagram. E
ient
lipping algorithm [2℄.
◮ Convert to other models of hyperboli
geometry : Poin
aré
disk, upper half spa
e, hyperboloid, Beltrami hemisphere, et
.
◮ Conformal (good for vizualizing) versus non-
onformal
(good for
omputing) models. (
onformal metri
G(x) = (x)I is s
aled identity metri
)

2014 Frank NIELSEN 5793b870 23/57

Hyperboli
Hyperboli
Voronoi diagram in Klein disk :

2014 Frank NIELSEN 5793b870 24/57

Hyperboli
Ane Voronoi diagram equivalent to power diagram.
Power distan
e : kx − pk2 − wp (additively weighted ordinary
Voronoi)

2014 Frank NIELSEN 5793b870 25/57

Hyperboli
5
ommon models of the abstra
t hyperboli
geometry
https://www.youtube.
om/wat
h?v=i9IUzNxeH4o
ACM Symposium on Computational Geometry (SoCG'14)

2014 Frank NIELSEN 5793b870 26/57

k-order hyperboli
Voronoi diagram
k = 1 k = 2
Ane diagram ⇒ equivalent to a power diagram [21℄
Wat
h k-order Voronoi :
https://www.youtube.
om/wat
h?v=i9IUzNxeH4o

2014 Frank NIELSEN 5793b870 27/57

Constrast fun
tion and dually at spa
e
◮ Convex and stri
tly dierentiable fun
tion F() admits a
Legendre-Fen
hel
onvex
onjugate F∗() :
F∗() = sup

(⊤ − F()), ∇F() = = (∇F∗)−1()
◮ Young's inequality gives rise to
anoni
al divergen
e [8℄ :
F()+F∗(′) ≥ ⊤′ ⇒ AF,F∗(, ′) = F() + F∗(′) − ⊤′
◮ Writing using single
oordinate system, get dual Bregman
divergen
es :
BF (p : q) = F(p) − F(q) − (p − q)⊤∇F(q)
= BF∗(q : p)
= AF,F∗(p, q) = AF∗,F (q : p)

2014 Frank NIELSEN 5793b870 28/57

Dis
riminant analysis exponential families
◮ Consider X1 ∼ EF (1), X2 ∼ EF (2) (or moment
parameterization i = ∇F(i ))
◮ Bayes' rule : Classify x from X1 i. P(x|1) P(x|2)
◮ Use bije
tion between exponential families and Bregman
divergen
es :
log p(x|) = −BF∗(t(x) : ) + F∗(t(x)) + k(x)
◮ MAP rule (w1 = w2 = 12
) :
BF∗(t(x) : 1) BF∗(t(x) : 2)
◮ Bregman bise
tor :
BiF∗(1, 2) = { |BF∗ ( : 1) = BF∗( : 1)}

2014 Frank NIELSEN 5793b870 29/57

Bregman dual bise
tors [3, 17, 20℄
Right-sided bise
tor : → Hyperplane (-hyperplane)
HF (p, q) = {x ∈ X | BF (x : p) = BF (x : q)}.
HF :
h∇F(p) − ∇F(q), xi + (F(p) − F(q) + hq,∇F(q)i − hp,∇F(p)i) = 0
Left-sided bise
tor : → -Hypersurfa
e (-hyperplane)
H′F(p, q) = {x ∈ X | BF (p : x) = BF (q : x)}
H′F
: h∇F(x), q − pi + F(p) − F(q) = 0

2014 Frank NIELSEN 5793b870 30/57

Visualizing Bregman bise
tors
Primal
oordinates Dual
oordinates
natural parameters expe
tation parameters
p
q
Source Space: Itakura-Saito
p(0.52977081,0.72041688) q(0.85824458,0.29083834)
D(p,q)=0.66969016 D(q,p)=0.44835617
Gradient Space: Itakura-Saito dual
p’(-1.88760873,-1.38808518) q’(-1.16516903,-3.43833618)
D*(p’,q’)=0.44835617 D*(q’,p’)=0.66969016
p’
q’

2014 Frank NIELSEN 5793b870 31/57

Spa
e of Bregman spheres and Bregman balls [3℄
Dual Bregman balls (bounding Bregman spheres) :
Ballr
F (c, r ) = {x ∈ X | BF (x : c) ≤ r}
and Balll
F (c, r ) = {x ∈ X | BF (c : x) ≤ r}
Legendre duality :
F (c, r ) = (∇F)−1(Ballr
F∗(∇F(c), r ))
Balll
Illustration for Itakura-Saito divergen
e, F(x) = −log x

2014 Frank NIELSEN 5793b870 32/57

Lifting/Polarity : Potential fun
tion graph F

2014 Frank NIELSEN 5793b870 33/57

Spa
e of Bregman spheres : Lifting map [3℄
F : x7→ ˆx = (x, F(x)), hypersurfa
e in Rd+1.
Hp : Tangent hyperplane at ˆp, z = Hp(x) = hx −p,∇F(p)i+F(p)
◮ Bregman sphere −→ ˆ with supporting hyperplane
H : z = hx − c,∇F(c)i + F(c) + r .
(// to Hc and shifted verti
ally by r )
ˆ = F ∩ H.
◮ interse
tion of any hyperplane H with F proje
ts onto X as a
Bregman sphere :
H : z = hx, ai+b → : BallF (c = (∇F)−1(a), r = ha, ci−F(c)+b)

2014 Frank NIELSEN 5793b870 34/57

Lifting with Itakura-Saito potential fun
tion

2014 Frank NIELSEN 5793b870 35/57

Spa
e of Bregman spheres [3℄
◮ Union/interse
tion of Bregman spheres from representational
polytope [3℄
◮ Radi
al axis of two Bregman balls is an hyperplane :
Appli
ations to Nearest Neighbor sear
h trees like Bregman
ball trees or Bregman vantage point trees [23℄.

2014 Frank NIELSEN 5793b870 36/57

Spa
e of spheres : Minimum en
losing ball [14, 24℄
To a hyperplane H = H(a, b) : z = ha, xi+b in Rd+1,
orresponds
a ball = Ball(c, r ) in Rd with
enter c = ∇F∗(a) and radius :
r = ha, ci−F(c)+b = ha,∇F∗(a)i−F(∇F∗(a))+b = F∗(a)+b
sin
e F(∇F∗(a)) = h∇F∗(a), ai − F∗(a) (Young equality)
SEB : Find halfspa
e H(a, b)− : z ≤ ha, xi + b that
ontains all
lifted points :
min
a,b
r = F∗(a) + b,
∀i ∈ {1, ..., n}, ha, xi i + b − F(xi ) ≥ 0.
⇒ Convex Program (CP) with linear inequality
onstraints
⇒ F() = F∗() = 12
x⊤x : CP → Quadrati
Programming
(QP) [4℄.

2014 Frank NIELSEN 5793b870 37/57

InSphere predi
ates wrt Bregman divergen
es [3℄
Impli
it representation of Bregman spheres/balls :
◮ Is x inside the Bregman ball dened by d + 1 support
points ?
InSphere(x; p0, ..., pd ) =

1 ... 1 1
p0 ... pd x
F(p0) ... F(pd ) F(x)

◮ InSphere(x; p0, ..., pd ) is negative, null or positive depending
on whether x lies inside, on, or outside .

2014 Frank NIELSEN 5793b870 38/57

Bregman Voronoi diagrams as minimization diagrams [3℄
A sub
lass of ane diagrams whi
h have all non-empty
ells .
Minimization diagram of the n fun
tions
Di (x) = BF (x : pi ) = F(x) − F(pi ) − hx − pi ,∇F(pi )i.
≡ minimization of n linear fun
tions :
Hi (x) = (pi − x)T∇F(qi ) − F(pi )
⇐⇒
Wat
h https://www.youtube.
om/wat
h?v=L7v1wuN_9Wg

2014 Frank NIELSEN 5793b870 39/57

Bregman Voronoi from Power diagrams [15, 20℄
Any ane diagram
an be built from a power diagram.
(power diagrams dened in full spa
e Rd )
◮ Power distan
e of x to Ball(p, r ) : ||p − x||2 − r 2.
◮ Laguerre diagram : minimization diagram of
Di (x) = ||pi − x||2 − r 2
i
◮ Power bise
tor of Ball(pi , ri ) and Ball(pj , rj )= radi
al
hyperplane :
2hx, pj − pi i + ||pi ||2 − ||pj ||2 + r 2
j − r 2
i = 0.
Universality : Ane Bregman Voronoi diagram ⇔ Power diagram

2014 Frank NIELSEN 5793b870 40/57

Ane Bregman Voronoi diagrams as power diagrams
Equivalen
e : B(∇F(pi ), ri ) with
r 2
i = h∇F(pi ),∇F(pi )i + 2(F(pi ) − hpi ,∇F (pi )i
(imaginary radii shown in red, WLOG. ri ≥ 0 by shifting)
Some
ells may be empty in the Laguerre diagram (power diagram)
but not in the Bregman diagram
http://www.
sl.sony.
o.jp/person/nielsen/BVDapplet/

2014 Frank NIELSEN 5793b870 41/57

Bregman dual Delaunay triangulations
Delaunay Exponential Hellinger-like
◮ empty Bregman sphere property,
◮ geodesi
triangles.
BVDs extends Eu
lidean Voronoi diagrams with similar
omplexity/algorithms.

2014 Frank NIELSEN 5793b870 42/57

Riemannian tensor and orthogonality
Dually at spa
e.
◮ F()
onvex on yields
onvex
onjugate F∗() on H
◮ Same Riemannian tensor
an be expressed in both
oordinate
systems :
g() = ∇2F(), g∗() = ∇2F(), g((X))g∗((X)) = I , ∀X
Same innitesimal length element ds2 = (ds∗)2.
◮ Two
urves
1(t) and
2(t) are orthogonal at
Q =
1(t0) =
2(t0) i.
h ˙
1(t0), ˙
2(t0)ig((Q)) = h ˙
1(t0), ˙
2(t0)i∗g
∗((Q)) = 0
◮ Geodesi

(P,Q) and dual geodesi

∗(P,Q) (and geodesi
segments
[P,Q] and
∗[P,Q])

2014 Frank NIELSEN 5793b870 43/57

Orthogonality
3-point property (generalized law of
osines) :
BF (p : r ) = BF (p : q) + BF (q : r ) − (p − q)T (∇F(r ) − ∇F(q)))
P
Q
R

[P,Q]

[Q,R]

(P,Q) orthogonal to
∗(Q, R) i.
BF (p : r ) = BF (p : q) + BF (q : r )
Equivalent to hp − q, r − qi = 0
Extend Pythagoras' theorem

(P,Q) ⊥
∗(Q, R)

2014 Frank NIELSEN 5793b870 44/57

Dually orthogonal Bregman Voronoi Triangulations
Ordinary Voronoi diagram is perpendi
ular to Delaunay
triangulation. (Vor k-fa
e ⊥ Del d − k-fa
e)
Dual line segment geodesi
s :

(P,Q) = { = p + (1 − )q | ∈ [0, 1]}

∗(P,Q) = { = p + (1 − )q | ∈ [0, 1]}
Bise
tors :
Bi(p, q) : hx, q − pi + F(p) − F(q) = 0
Bi(p, q) : hx, q − pi + F∗(p) − F∗(q) = 0
Dual orthogonality :
Bi(p, q) ⊥
∗(P,Q)

(P,Q) ⊥ Bi(p, q)

2014 Frank NIELSEN 5793b870 45/57

Dually orthogonal Bregman Voronoi Triangulations
Bi(p, q) ⊥
∗(P,Q)

(P,Q) ⊥ Bi(p, q)

2014 Frank NIELSEN 5793b870 46/57

MAP rule and additive Bregman Voronoi diagrams
Optimal MAP de
ision rule :
map(x) = argmaxi∈{1,...,n}wipi (x)
= argmaxi∈{1,...,n} − B∗(t(x) : i ) + log wi ,
= argmini∈{1,...,n}B∗(t(x) : i ) − log wi
⇒dual Bregman Voronoi with additive weights (ane
diagram).
Verti
al proje
tion of the interse
tions of the lifted half-spa
es on
the potential F∗ shifted by the weights −log wi .

2014 Frank NIELSEN 5793b870 47/57

Relative entropy for exponential families [18℄
◮ Kullba
k-Leibler divergen
e (
ross-entropy minus entropy) :
KL(P : Q) =
Z
p(x) log
p(x)
q(x)
dx ≥ 0
=
Z
p(x) log
1
q(x)
dx
| {z }
H×(P:Q)+c
−
Z
p(x) log
1
p(x)
dx
| {z }
H(p)=H×(P:P)−c
= F(Q) − F(P) − hQ − P,∇F(P)i
= BF (Q : P) = BF∗(P : Q)
Bregman divergen
e BF dened for a stri
tly
onvex and
dierentiable fun
tion up to some ane terms.
◮ Proof KL(P : Q) = BF (Q : P) follows from
X ∼ EF () =⇒ E[t(X)] = ∇F() =

2014 Frank NIELSEN 5793b870 48/57

Voronoi diagrams in information geometry: Statistical Voronoi diagrams and their applications

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Voronoi diagrams in information geometry: Statistical Voronoi diagrams and their applications

Similar to Voronoi diagrams in information geometry: Statistical Voronoi diagrams and their applications (20)

Recently uploaded

Recently uploaded (16)