4. Introduction
2014-03-05 Bellet, A., Habrard, A., and Sebban, M. A Survey on Metric Learning for Feature Vectors and Structured Data, 2013 4
Bellet, A., Habrard, A., and Sebban, M. A Survey on Metric Learning for Feature Vectors and Structured Data, 2013
5. Categories of distance metric learning
•Superviseddistancemetriclearning
•Constraintsorlabelsgiventothealgorithm
•Equivalenceconstraintswherepairsofdatapointsthatbelongtothesameclasses(semantically-similar)
•Inequivalenceconstraintswherepairsofdatapointsthatbelongtothedifferentclasses(semantically-dissimilar)
•localadaptivesuperviseddistancemetriclearning,NCA,RCA
•Unsuperviseddistancemetriclearning
•Dimensionalityreductiontechniques
•Linearmethods(PCA,MDS)andnon-linearmethods(ISOMAP,LLE,LE)
•Kernelmethodstowardslearningdistancemetrics
•Kernelalignment,extensionworkoflearningidealizedkernel
2014-03-05 Liu Yang, An Overview of Distance Metric Learning, 2007 5
Liu Yang, Distance Metric Learning: A Comprehensive Survey, 2005
Liu Yang, An Overview of Distance Metric Learning, 2007
6. Categories of distance metric learning
•Maximummarginbaseddistancemetriclearning
•Formulatedistancemetriclearningasaconstrainedconvexprogrammingproblem,andattempttolearncompletedistancemetricfromtrainingdata
•Expensivecomputation,overfittingproblemwhenlimitedtrainingsamplesandlargefeaturedimension
•Largemarginnearestneighborclassification(LMNN),SDPmethodstosolvethekernelizedmarginmaximization
2014-03-05 Liu Yang, An Overview of Distance Metric Learning, 2007 6
Liu Yang, Distance Metric Learning: A Comprehensive Survey, 2005
Liu Yang, An Overview of Distance Metric Learning, 2007
7. Categories of distance metric learning
2014-03-05 Liu Yang, An Overview of Distance Metric Learning, 2007 7
Liu Yang, An Overview of Distance Metric Learning, 2007
8. Large Margin Nearest Neighbor Classification (LMNN)
KilianWeinberger,etal.Distancemetriclearningforlargemarginnearestneighborclassification.NIPS,2006.
2014-03-05 Kilian Weinberger, et al. Distance metric learning for large margin nearest neighbor classification. NIPS, 2006. 8
9. Motivation –better KNN
•KNN:simplenon-parametricclassificationmethod(Alsocanbeusedforclustering,regression,anomalydetection,andetc.)
•푘isuser-definedconstant,andanunlabeledvectorisclassifiedbyassigningthelabelwhichismostfrequentamongthe푘trainingsamplesnearesttothatunlabeledvector
•Drawback:whentheclassdistributionisskewed,basic“majorityvoting” classificationmaynotworkswell.
2014-03-05 Kilian Weinberger, et al. Distance metric learning for large margin nearest neighbor classification. NIPS, 2006. 9
18. Results
2014-03-05 Kilian Weinberger, et al. Distance metric learning for large margin nearest neighbor classification. NIPS, 2006. 18
19. GBLMNN: Non-linear method for LMNN
•Buildingnon-linearmappingformetriclearning
•GBRT:GradientBoostingRegressionTree
•Code:http://www.cse.wustl.edu/~kilian/code/code.html
2014-03-05 Kedem, D., Xu, Z., & Weinberger, K. (n.d.). Gradient Boosted Large Margin Nearest Neighbors, (1), 10–12. 19
20. Large Margin Component Analysis (LMCA)
Torresani,L.,&Lee,K.(2007).Largemargincomponentanalysis.AdvancesinNeuralInformationProcessing
2014-03-05 Torresani, L., & Lee, K. (2007). Large margin component analysis. Advances in Neural Information Processing 20
21. Problem: large number of features
•Inproblemsinvolvingthousandsoffeatures,distancemetriclearningcannotbeused
•Highcomputationcomplexity
•Overffitingproblem
•Previousworkshavereliedonatwo-stepsolution
•Applydimensionreductionalgorithm(e.g.PCA)
•Learnametricintheresultinglow-dimensionalsubspace
•LMCA:providebettersolutionforhighdimensionalproblem
2014-03-05 Torresani, L., & Lee, K. (2007). Large margin component analysis. Advances in Neural Information Processing 21
22. Linear Dimensionality Reduction for KNN (1/3)
•Recall:costfunctionofmetricfunction
•Inpreviouswork,Lwas퐷×퐷matrix
•Idea:learnnonsquarematrixsize푑×퐷,푤푖푡ℎ푑≪퐷
•Problem:costfunction휖퐌,푤ℎ푒푟푒퐌=퐋⊤퐋isnolongerconvex
•Mhaslowrankd,notD
2014-03-05 Torresani, L., & Lee, K. (2007). Large margin component analysis. Advances in Neural Information Processing 22
23. Linear Dimensionality Reduction for KNN (2/3)
•Idea:optimizetheobjectivefunctiondirectlywithrespecttothenonsquarematrixL
•Although,equationisnon-convex,itisbetterthanminimizingtheobjectivefunctionwithrespecttoLratherthanwithrespecttotherank- deficientD×DmatrixM
•Sincethisoptimizationinvolvesonly푑∙퐷ratherthan퐷2unknowns,itcanreducetheriskofoverffiting
•“theoptimalrectangularmatrixLcomputedwithourmethodautomaticallysatisfiestherankconstraintsonMwithoutrequiringthesolutionofdifficultconstrainedminimizationproblems”
2014-03-05 Torresani, L., & Lee, K. (2007). Large margin component analysis. Advances in Neural Information Processing 23
24. Linear Dimensionality Reduction for KNN (3/3)
•Minimizecostfunctionusinggradient-basedoptimizers
•“Wehandlethenon-differentiabilityofh(s)ats=0,byadoptingasmoothhingefunctionasin[8]”
2014-03-05 Torresani, L., & Lee, K. (2007). Large margin component analysis. Advances in Neural Information Processing 24
[8] J. D. M. Rennieand N. Srebro. Fast maximum margin matrix factorization for collaborative prediction.
In Proceedings of the 22nd International Conference on Machine Learning (ICML), 2005.
25. Nonlinear Feature Extraction (1/2)
•“Ourapproachlearnsalow-rankMahalanobisdistancemetricinahighdimensionalfeaturespaceF,relatedtotheinputsbyanonlinearmap 휙:푅퐷→퐹”
•kiskernelfunctioncomputedbythefeatureinputproducts
•LisnowatransformationfromthespaceFintoa푅푑(푑≪퐷)
•
2014-03-05 Torresani, L., & Lee, K. (2007). Large margin component analysis. Advances in Neural Information Processing 25
26. Nonlinear Feature Extraction (2/2)
•
•
•
•UpdateΩ←(Ω−휆Γ)untilconverge
•Forclassification,weprojectpointsontothelearnedlow-dimspacebyexploitingthekerneltrick:퐿흓=Ω퐤
2014-03-05 Torresani, L., & Lee, K. (2007). Large margin component analysis. Advances in Neural Information Processing 26
27. Results –High dimensional dataset
2014-03-05 Torresani, L., & Lee, K. (2007). Large margin component analysis. Advances in Neural Information Processing 27
28. Results –low dimensional dataset
2014-03-05 Torresani, L., & Lee, K. (2007). Large margin component analysis. Advances in Neural Information Processing 28