MLPI Lecture 1: Maths for Machine Learning

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Lecture'1
Maths&for&Machine&Learning
Dahua%Lin
The$Chinese$University$of$Hong$Kong
1

Roadmap
Mathema'cs!lies!at!the!heart!of!machine!learning!
research.!
Complete(coverage(of(all(these(subjects(is(obviously(...

Concepts)to)Be)Covered
• Basics'of'topology
• metrics
• open'and'closed-sets
• con.nuous-func.on
• compact-space
3

Concepts)to)Be)Covered)(cont'd)
• Basics'of'func,onal'analysis
• norm,'inner'product
• Banach'space,'Hilbert'space
• func5...

Concepts)to)Be)Covered)(cont'd)
• Basics'of'modern'probability'theory:'
• measure'space
• Lebesgue'integra0on
• random'var...

Metrics
Measurement*of*distances*or*devia*on*lies*at*the*
heart*of*many*learning*problems.*
6

Metrics((Deﬁni-on)
• "is"called"a"metric,"if"it"sa-sﬁes:
• (non*nega-vity)"
• (coincidence2axiom)"
• (symmetry)"
• (triang...

Metrics((Proper-es)
• The%triangle)inequality%can%be%further%generalized:
!
• A#set# #together#with#a#metric# #deﬁned#ther...

Examples)of)Metrics
• Euclidean*metric:
• Rec$linear*metric:
!
9

Metrics(Induced(by(Transform
• One%can%deﬁne%a%metric%over%an%arbitrary%set%$S$%
through%an%injec&ve(map% %to%a%metric%
sp...

Topology'Deﬁned'by'Metrics
• Topological)space"is"a"more"general"concept"than"
metric)space
• With"metrics,"topological)co...

Interior
• Open%ball"of"radius" :"
• "is"an"interior'point"of" "iﬀ" ."
• The"interior"of" ,"denoted"by" "or" ,"is"the"
set...

Boundary
• "is"a"boundary)point"of" ,"iﬀ"for"any" ," "
overlaps"with"both" "and" ."
• The"boundary"of" ,"denoted"by" ,"is"...

Open%and%Closed%Sets
• Consider*a*metric*space* ,*and*a*subset*
:
• *is*called*an*open%set,*if*
• *is*called*a*closed%set,...

Proper&es(of(Open(and(Closed(Sets
• The%union%of%arbitrary%collec-ons%of%open%sets%is%
open.
• The%intersec-on%of%ﬁnitely+...

Ques%ons
• Consider*an*arbitrary*metric*space* ,*are* *and*
*open*or*closed?
• Let* *be*a*ﬁnite*metric*space*and* ,*is* *
...

Closure
Let$ $be$a$subset$of$a$metric$space$ :
• The%closure%of% ,%denoted%by% %or% ,%is%deﬁned%
to%be%the%intersec3on%of%...

Closure((Examples)
• .
• .
18

Convergence
• Let% %be%a%sequence%in%a%metric%space%
,%then% %is%called%a%limit%of%this%sequence%
if
• A#sequence#is#said#...

Cauchy'Sequence
• A#sequence# #is#called#a#Cauchy'sequence#if#
!
• A#convergent)sequence#must#be#a#Cauchy)sequence
• Ques%...

Mo#va#ng(Example
• Consider*a*real*valued*sequence*
• We$have$learned$in$Calculus$that$ $as$
.
• Note$that$ $is$a$ra5onal$...

Complete(Metric(Space
• Convergence)is)not)an)intrinsic)property)of)a)sequence,)
which)also)depends)on)the)space)in)which)...

Closedness(and(Completeness
Let$ $be$a$subset$of$a$complete$metric$space$ :
• Being'closed'means'"containing-all-the-limit...

Bounded
• The%diameter%of% ,%denoted%by% ,%is%deﬁned%to%
be%the%largest%distance%between%points%in% ,%as
• "is"said"to"be"...

Compactness
• In$elementary$Calculus,$we$learned$that$"every&
bounded&sequence&in& &has&a&convergent&
subsequence".$
• Doe...

Compactness+(cont'd)
Consider)a)metric)space) :
• "is"called"compact"iﬀ"either"of"the"following"holds:
• Any"open"cover"of...

Compactness+(cont'd)
• Compact)sets)are)always)closed'and'bounded.
• But,)the)converse)is)generally'false.
• )is)compact)i...

Con$nuous'Func$on
• "is"called"con$nuous,"iﬀ"either"of"the"
following"holds:"
• (Topological)" "is"
open"whenever" "is"ope...

Con$nui$es)of)Various)Levels
• "is"called"uniformly*con,nuous,"if"
• "is"called"Lipschitz)con,nuous,"if"
!
• Lipschitz)con...

Con$nuous'Funcs'on'Compact'Set
• Let% %be%a%con+nuous%func+on,%then%
%is%compact%whenever% %is%
compact.
• Let% ,%then% %a...

Dense%Sets
• A#subset# #is#said#to#be#dense%in% ,#if# .
• A#subset# #is#dense%in% ,#iﬀ#either#of#the#following#
holds:
• E...

Separable(Space
• "is"called"a"separable(space,"if" "contains"a"
countable"dense"subset.
• Let" "be"a"con3nuous"func3on"on...

Things'become'much'more'
interes0ng'when'vector'spaces'meet'
with'metrics.
33

Norms
• Consider*a*vector'space* ,*a*func0on* *
is*called*a*norm*if:
•
• *iﬀ* *
• .
• .
34

Banach&Spaces
• A#vector#space#together#with#a#norm,#as# ,#
is#called#a#normed'space.#
• A#normed'space#is#always#a#metric...

Proper&es(of(Norms
• The%norm%func*on% %is%Lipschitz%
con*nuous:
• The%metric% %induced%by%the%norm%has:
• transla'on)inva...

!Norm
Consider)a)real)vector)space) .)For)each) ,)we)
can)deﬁne) 6norm)as:
It#can#be#easily#veriﬁed#that# #is#a#norm.#
37

!Norm
When% ,% 'norm%is%s-ll%a%norm,%deﬁned%as
We#will#extend#these#norms#to#the#space&of&func+ons.
38

Convergence)of)Vector)Sequences
• Let% %be%a%sequence%of%vectors%in%a%Banach%
space,%we%say% %converges%to% %if% %
as% .
•...

Convergence)of)Series
• Consider*an*inﬁnite*series* *
• Let* .*The*series* *is*said*to*be*
convergent*if*the*sequence* *co...

Basis
• In$elementary$linear$algebra,$we$learned$that$a$basis$
of$a$vector$space$is$a$linearly*independent$set$of$
vectors...

Basis%of%Banach%Space
• Let% %be%a%sequence%in%a%Banach%space% ,%it%is%
called%a%(Schauder)+basis%of% %if%for%each% ,%
the...

Basis%of%Banach%Space%(cont'd)
• It$took$about$forty$years$to$get$the$answer$33$No.$
• Enﬂo$constructed$a$separable$Banach...

Inner%Products
• An$inner$product$on$a$real$vector$space$ $is$a$
mapping$ $that$sa5sﬁes:
•
• $iﬀ$
• (symmetry)$
• (bilinea...

Hilbert(Spaces
• A#vector#space#together#with#an#inner#product#is#
called#an#inner%product%space.
• An#inner%product%space...

Euclidean*Space
• "is"a"Hilbert(space"with"the"inner"product"deﬁned"
by" ,"which"induces"the" 5norm"
and"Euclidean(metric....

Proper&es(of(Inner(Products
• (Parallelogram*equality)"
"##"this"only"
holds"for"norms"induced"by"inner"products.
• (Cauch...

Orthogonality
• "are"said"to"be"orthogonal"to"each"other,"
denoted"by" ,"if" .
• A"subset" "is"called"an"orthogonal)set"if...

Projec'on
• Let% %and% ,%then%the%distance%between% %
and% %is%deﬁned%to%be% .
• Let% %be%a%non2empty%convex%closed%subset...

Projec'on)(cont'd)
Let$ $be$a$Hilbert$space$and$ $be$a$closed$subspace:
• Given' 'and' ,'then' ,'
meaning'that' .
Let$ $be...

Orthogonal*Complement
• A#vector#space# #is#called#the#direct'sum#of#two#
subspaces# #and# ,#denoted#by# ,#if#each#
#can#b...

Orthonormal*Basis*of*Hilbert*Space
• Let% %be%a%Hilbert%space,%a%subset% %is%called%a%
total%subset%of% %if% %is%dense.
• ...

Orthonormal*Basis*of*Hilbert*Space
• (Parseval)theorem)"An"orthonormal"sequence" "is"
total"if"and"only"if
• In$this$case,...

Orthonormal*Basis*of*Hilbert*Space*
(cont'd)
• Every'Hilbert'space'has'a'total%orthonormal%set.'
• Every'separable'Hilbert...

Spaces
55

Linear'Operators'and'Func1onals
• Let% %and% %be%two%linear%spaces,%a%func5on:%
%is%called%a%linear'operator%if%it%preserv...

Bounded'Operators
Let$ $be$a$linear$operator$on$Banach$spaces:
• "is"said"to"be"bounded,"if"
.
• The"(operator)-norm"of" "...

Bounded'Operators'(cont'd)
• Given'a'linear'operator' ,'the'following'statements'
are'equivalent:
• 'is'bounded
• 'is'con;...

Space&of&Bounded&Operators
• All$bounded$operators$ $form$a$normed)
space,$denoted$by$ .$
• When$ $is$a$Banach$space,$ $is...

Dual%Spaces
• All$bounded$func)onals$ $form$a$normed)
space,$called$the$dual)space$of$ ,$denoted$by$ ,$
where$the$norm$is$...

Func%on'Space'and'Uniform'Norm
• The%set%of%all%con$nuous%real%valued%func2ons%
deﬁned%on%a%compact+space% %forms%a%normed...

Ques%ons
• Let% %be%a%subspace%of% %that%
comprises%all%con$nuously)diﬀeren$able)func$ons%on%
.%Let% .%We%deﬁne%a%func8ona...

Representa)on+of+Func)onals
• Every'Hilbert'space'is'isomorphic'to'its'dual%space.
• (Riesz's%theorem)'Every'bounded'linea...

Representa)on+of+Func)onals+
(cont'd)
• In$a$separable$Hilbert$space,$ $can$be$expressed$as$
a$series:
64

Measure'Theory
• Measure'theory"studies"assigning'values'to'subsets.
• Measure"theory"is"the"corner"stone"of"many"math"
su...

Lengths(of(Real(Subsets
• Intui'vely,-the-measure-of-a-set-can-be-intepreted-
as-the-size,-e.g.-the-length-of-an-interval....

Lengths(of(Real(Subsets((cont'd)
• Sizes'(e.g.'lengths,'areas,'and'volumes)'are'countably-
addi0ve.
• Let' 'be'a'collec;on...

The$Vitali$Set
• Let's'consider'a'very'interes1ng'subset'of' ,'
called'Vitali&set:
• We'say' 'and' 'are'equivalent,'if' .
...

The$length$of$Vitali$Set
• Enumerate*all*ra,onal*numbers*within* *as*
,*and*let* .*Obviously,*
*are*disjoint
• Let* ,*then...

Ring%of%Sets
• ,#a#nonempty#collec.on#of#subsets#of# ,#is#called#a#
ring,#if#it#is#closed#under#union#and#set*diﬀerence.
•...

!algebra
• A#ring# #is#called#a#algebra,#if#it#is#also#closed#under#
complement.
• An#algebra#must#contain# .
• An#algebra...

Measure'and'Measure'Space
• A#countably#addi/ve#func/on# #from#a# 5algebra# #
into# #is#called#a#measure.#
• The#triple# #...

Proper&es(of(Measures
Consider)a)measure'space) :
• .
• .
• Let& &be&disjoint,&then&
.
73

Con$nuity)of)Measures
.
.
74

Finite&and& )finite&Measures
• "is"called"a"finite&measure"if" .
• "is"called"a" ,finite&measure"if" "is"covered"by"
countabl...

Lengths(of(Open(Intervals
• The%length%of%an%open%interval%is%deﬁned%as%
.
• The%the%length%of%a%ﬁnite%union%of%disjoint%o...

Lengths(of(Open(Intervals((cont'd)
• The%collec)on%of%all%ﬁnite%unions%of%disjoint%open%
intervals%cons)tutes%a%ring.
• Th...

Generated( )algebra
• Let% %be%a%collec+on%of%subsets.%Then%the%smallest%
4algebra%that%contains% %is%called%the% !algebra...

Borel& 'algebra
• Let% %be%a%metric'space.%The% +algebra%generated%by%
the%open%subsets%of% %is%called%the%Borel' .algebra...

Measure'Extension
• (Hahn–Kolmogorov.theorem):"Let" "be"a"ring"that"
covers" ," "be"a"pre4measure,"then" "
can"be"extended...

Complete(Measure(Space
We#are#not#done#yet.#Given#a#measure#space#
:
• A#subset# #is#called#a#null$set#if#it#is#contained#...

Lebesgue'Measure
• The%Borel&measure&space%is%generally%not%complete.%
• The%comple.on%of%the%Borel&measure&space%is%calle...

Lebesgue'Integral'.'Indicator'
Func4ons
To#derive#the#Lebesgue'integral,#we#begin#with#simple#
func7ons#and#then#extend#th...

Lebesgue'Integral'.'Simple'Func5ons
• Simple(func-ons(00(linear(combina-ons(of(indicator(
func-ons:
84

Lebesgue'Integral'.'Nonnega1ve'
Func1ons
• Let% %be%a%non#nega've%func,on%on% ,%then%we%
deﬁne
85

Lebesgue'Integral'.'Signed'Func4ons
• Signed(func,on:(decompose( (as( ,(
with( (and( ,(then
• When&both& &and& &are&inﬁnit...

Proper&es(of(Lebesgue(Integral
• "is"a"linear"func-onal.
• If" "then" "
• a"predicate"holds"almost'everywhere"means"that"i...

Monotone&Convergence&Theorem
Let$ $be$a$sequence$of$non#nega've$func.ons,$and$
$pointwisely,$then$
88

Dominated*Convergence*Theorem
If# #pointwisely,# #is#dominated#by# ,#i.e.#
,#and# ,#then#
89

Integral)with)Coun0ng)Measure
• The%coun%ng'measure%over%a%ﬁnite%or%countable%
space% %is%deﬁned%as%the%cardinality%of%sub...

Integral)with)Lebesgue)Measure
• Let% %be%a%func,on%over% ,%then%when% %is%
Riemann'integrable,% %must%be%Lebesgue'integra...

!space
• For%any% ,%the% %space%of% ,%denoted%by%
,%is%the%vector%space%comprised%of%all%
measurable%func8ons%on% %such%th...

!space!(cont'd)
• Func&ons)that)are)equal)almost'everywhere)are)
indis&nguishable)by)integra&on.)Let) )denotes)
the)vector...

!Space!and!Dual
• The%dual%space%of% %is%isomorphic%to% %with%
.%
• For%each%bounded%func8onal%on% ,%there%
exists% :%
94

!Hilbert!Space
• "is"a"Hilbert"space,"where"the"inner"product"is"
deﬁned"as
• The%dual%space%of% %is% .
95

Absolute)Con,nuity
Let$ $and$ $be$two$measures$over$ :
• "is"said"to"be"absolutely*con-nuous"with"respect"to"
,"denoted"by...

Lebesgue'Decomposi.on
Let$ $and$ $be$two$measures$over$ :
There%exists%a%unique%decomposi3on%of% %as%
!
such%that% %and% ....

Radon&Nikodym,Theorem
Let$ $be$a$ﬁnite$measure$that$is$absolutely*con-nuous$
with$respect$to$ .$Then$there$exists$an$essen...

Event&Spaces
• The% &algebra% %can%be%interpreted%as%an%event%
space.%Each%element% %is%an%event.
• % %both% %and% %happen...

Probability*Measure
• A#probability*measure# #is#a#measure#on# #with#
.
• #can#be#considered#as#the#probability#of#the#
ev...

Independence
• Two%events% %and% %are%said%to%be%independent%if%
• We$say$two$collec-ons$of$events$ $and$ $are$
independen...

Random'Variables
• A#(real&valued)+random+variable# #is#deﬁned#to#be#a#
measurable+func4on# .
• Each#random#variable# #ind...

Expecta(on
• The%expecta'on%of%a%random%variable% %is%deﬁned%
to%be
!
• The%expecta'on%is%a%bounded-linear-func'onal.
103

Probability*and*Law
• The%probability%that%the%value%of% %falling%in% %is%
then%given%by%
."
• The%func*on% ,%which%maps%e...

Probability*Density
• The%Radon+Nikodym%density%of% %with%respect%
to%the%Lebesgue%measure%over% ,%denoted%by% ,%is%
calle...

(Seman'c)*Correspondence
• "algebra) )event)space
• sub" "algebra) )a)family)of)events
• measure) ) )probability)func6on
•...

Convex'Analysis
• Convex(op*miza*on(plays(a(crucial(role(in(many(
machine(learning(problems
• Prof.(Stephen'Boyd(has(a(ver...

Convex'Analysis'(cont'd)
• Important*concepts
• convex'set
• convex'func,on
• conjugate'func,on
• Lagrange'dual
• strong'd...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Lecture 5: Variational Estimation a... by Dahua Lin 964 views
MLPI Lecture 0: Overview by Dahua Lin 1103 views
MLPI Lecture 4: Graphical Model and... by Dahua Lin 1064 views
MLPI Lecture 2: Monte Carlo Methods... by Dahua Lin 2203 views
MLPI Lecture 3: Advanced Sampling T... by Dahua Lin 1329 views

View on Slideshare

1 of 108 Ad

View on Slideshare

1 of 108 Ad