Parikshit Ram – Senior Machine Learning Scientist, Skytree at MLconf ATL
Max-kernel search: How to search for just about anything? Nearest neighbor search is a well studied and widely used task in computer science and is quite pervasive in everyday applications. While search is not synonymous with learning, search is a crucial tool for the most nonparametric form of learning. Nearest neighbor search can directly be used for all kinds of learning tasks — classification, regression, density estimation, outlier detection. Search is also the computational bottleneck in various other learning tasks such as clustering and dimensionality reduction. Key to nearest neighbor search is the notion of “near”-ness or similarity. Mercer kernels form a class of general nonlinear similarity functions and are widely used in machine learning. They can define a notion of similarity between pairs of objects of any arbitrary type and have been successfully applied to a wide variety of object types — fixed-length data, images, text, time series, graphs. I will present a technique to do nearest neighbor search with this class of similarity functions provably efficiently, hence facilitating faster learning for larger data.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Parikshit Ram – Senior Machine Learning Scientist, Skytree at MLconf ATL
Similar to Parikshit Ram – Senior Machine Learning Scientist, Skytree at MLconf ATL (20)
More from MLconf
More from MLconf (20)
Recently uploaded
Recently uploaded (20)
Parikshit Ram – Senior Machine Learning Scientist, Skytree at MLconf ATL
- 1. Max-kernel search How to search for just about anything? Parikshit Ram
- 2. Similarity search q ● Set of objects ● Query R ● Similarity function 1
- 4. Drug discovery 3 http://fineartamerica.com
- 6. Similarity search is ubiquitous ● Machine learning ● Computer vision ● Theory ● Databases ● Information retrieval ● Web application ● Collaborative filtering ● Scientific computing 5
- 9. Search-based classification 6 k-nearest-neighbor classification/regression
- 10. Search-based classification 7 “RomCom fan”
- 11. Search-based classification 7 “Kids movie fanatic”
- 13. 9
- 14. Search-based ML Advantage ● nonparametric - lets the data speak ● no need to train complex models Key ingredient ● notion of similarity (domain/data-specific) Main challenge: efficiency ● Sheer size of the data ● Varied data types 10
- 15. Properties of similarity functions 11 ● symmetry OR
- 16. 11 3 1 The dissimilarity is the size of the set-theoretic difference
- 17. Properties of similarity functions 11 ● symmetry ● self-similarity OR OR
- 18. 11 We do not really care about this.
- 19. Properties of similarity functions 11 ● symmetry ● self-similarity OR OR
- 20. 12
- 21. 12
- 22. 12 Metrics used everywhere
- 23. 12 Metrics used everywhere
- 24. 12 Bregman divergences widely used for distributions Mercer kernels widely used in ML for variety of objects and problems ??? not quite explored in search or ML Metrics used everywhere
- 25. Breadth of Kernel Functions Objects Kernel Functions Images linear, polynomial, Gaussian, Pyramid match Documents cosine Sequences p-spectrum kernel, alignment score Trees subtree, syntactic, partial tree Graphs random walk Time series cross-correlation, dynamic time-warping Natural Lang. convolution, decomposition, lexical semantic 13
- 26. What is a Kernel Function? In words A pairwise symmetric function ● Correlation in a richer but hidden feature space ● Cannot access the hidden space Object space Hidden space Hidden mapping 14
- 27. Max-kernel Search Find the object in R most similar to q with respect to a kernel 15
- 28. Existing methods ● Brute-force (parallel/distributed) ○ Domain-specific optimizations ● Coerce data to use metrics ○ Only approximate No standard search tools! 16
- 29. Understanding kernels If two objects equally similar to each other then they are equally similar to the query q 17
- 30. IF 17 Understanding kernels THEN
- 31. 18 Indexing our collection
- 32. 18 Indexing our collection
- 33. Multi-resolution index in O( n log n ) time p 18 Indexing our collection Cover Tree (BKL 2006)
- 34. How to Search with this Index? 19 q p
- 35. How to Search with this Index? 19 q p p' p''
- 36. How to Search with this Index? q p p'' p' 19
- 37. How to Search with this Index? q p p'' p' 19
- 38. How to Search with this Index? q p p'' p' Safely ignore a large chunk (potentially millions) 19
- 40. Results: Efficiency 10000x ● Widely applicable algorithm ● Performance data/kernel-dependent 10x Improvement 20
- 41. Results: Sublinear Query Time Improvement Object set size Bigger data implies bigger efficiency gains 21
- 42. Can We Prove it? What Makes Search Hard? Thm. For a set R of n objects, the query time is ● expansion constant ○ the distribution of the data ● directional concentration constant ○ the distribution of a kernel-induced transformation of the data 22
- 43. Endnote ● Search is an essential tool for ML ● Exploring different types of similarity functions increases the applicability and quality of search ● Kernels are widely applicable similarity functions ○ now we have provably fast max kernel search Code/tutorial for Fast Exact Max-Kernel Search 23 version 1.0.5 http://www.mlpack.org Ryan R. Curtin Email: pari@skytree.net