K-nearest neighbor search

GPU-based brute-force method

Experiments

Searching high-dimensional neighbors:
CPU-based tail...
K-nearest neighbor search

GPU-based brute-force method

Experiments

K-nearest neighbor search problem (kNN)
Input data
R...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Applications
Image processing
Tracking
Texture synth...
K-nearest neighbor search

GPU-based brute-force method

Experiments

State of the art
Brute-force method
Compute the dist...
K-nearest neighbor search

GPU-based brute-force method

Experiments

GPGPU
GPU: Graphics Processing Unit
Set of graphic p...
K-nearest neighbor search

GPU-based brute-force method

Experiments

GPU-based brute-force method
Brute-force kNN search ...
K-nearest neighbor search

GPU-based brute-force method

Experiments

GPU-based brute-force method
Brute-force kNN search ...
K-nearest neighbor search

GPU-based brute-force method

Experiments

GPU-based brute-force method
Brute-force kNN search ...
K-nearest neighbor search

GPU-based brute-force method

Experiments

GPU-based brute-force method: sorting algorithm
We h...
K-nearest neighbor search

GPU-based brute-force method

Experiments

GPU-based brute-force method: sorting algorithm
We h...
K-nearest neighbor search

GPU-based brute-force method

Experiments

GPU-based brute-force method: sorting algorithm
We h...
K-nearest neighbor search

GPU-based brute-force method

Experiments

GPU-based brute-force method: sorting algorithm
We h...
K-nearest neighbor search

GPU-based brute-force method

Experiments

GPU-based brute-force method: sorting algorithm
We h...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments
Goal
Decrease the computation time of th...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments
Goal
Decrease the computation time of th...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Synthetic data
Parameters
n: number of ...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Synthetic data
d=8

d=64

d=96

Methods...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Synthetic data
Influence of the paramete...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Synthetic data
Influence of the paramete...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Synthetic data
Influence of the paramete...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Finding similar patches in images
Goal
...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Finding similar patches in images
Goal
...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Finding similar patches in images
Goal
...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Finding similar patches in images
Data
...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Texture synthesis
Texture synthesis
Efr...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Texture synthesis
Efros and Leung

Garc...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Experiments: Texture synthesis
Data
Source image: 64...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Conclusion
We proposed a GPU-based brute-force kNN s...
K-nearest neighbor search

GPU-based brute-force method

Experiments

Questions

Thanks for your attention

Garcia, Nielse...
Upcoming SlideShare
Loading in …5
×

Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method (MIRAGE 2009)

264 views

Published on

Slides for the paper presented at MIRAGE 2009
http://link.springer.com/chapter/10.1007%2F978-3-642-01811-4_38?LI=true

Published in: Technology, Art & Photos
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
264
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method (MIRAGE 2009)

  1. 1. K-nearest neighbor search GPU-based brute-force method Experiments Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method Vincent Garcia and Frank Nielsen ´ Ecole Polytechnique, Palaiseau, France Mirage 2009 Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  2. 2. K-nearest neighbor search GPU-based brute-force method Experiments K-nearest neighbor search problem (kNN) Input data R = {r1 , r2 , · · · , rm }: set of m reference points in Rd Q = {q1 , q2 , · · · , qn }: set of n query points in Rd Problem: For a given query point q, find the k-nearest neighbors of q in R. Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  3. 3. K-nearest neighbor search GPU-based brute-force method Experiments Applications Image processing Tracking Texture synthesis Image retrieval Statistics Shannon entropy Kullback-Leibler divergence Other applications Physics Biology Finance Multimedia Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  4. 4. K-nearest neighbor search GPU-based brute-force method Experiments State of the art Brute-force method Compute the distances between q and the reference points Sort the distances and deduce the k-nearest neighbors Space-partitionning based methods Partition the reference points (kd-tree) Fast kNN search using this structure Locality sensitive hashing (LSH) Apply a set of hashing functions to the reference points (buckets) Apply the same hashing functions to q Compute the distances between q and the references points belonging to the same bucket Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  5. 5. K-nearest neighbor search GPU-based brute-force method Experiments GPGPU GPU: Graphics Processing Unit Set of graphic processors (up to 240 at 1.5 GHz) Dedicated graphics rendering device Specialized in highly parallel process GPGPU: General-Purpose computing on GPU Technique of using a GPU to perform computation in applications traditionally handled by the CPU NVIDIA CUDA (since Feb. 2007) OpenCL (specifications in Dec. 2008) Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  6. 6. K-nearest neighbor search GPU-based brute-force method Experiments GPU-based brute-force method Brute-force kNN search is highly parallelizable Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  7. 7. K-nearest neighbor search GPU-based brute-force method Experiments GPU-based brute-force method Brute-force kNN search is highly parallelizable Idea Propose a GPU implementation of the brute-force kNN search using NVIDIA CUDA Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  8. 8. K-nearest neighbor search GPU-based brute-force method Experiments GPU-based brute-force method Brute-force kNN search is highly parallelizable Idea Propose a GPU implementation of the brute-force kNN search using NVIDIA CUDA Input data: m reference points and n query points Kernels 1 Compute the m × n distances in parallel 2 Sort the n sets of distances in parallel Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  9. 9. K-nearest neighbor search GPU-based brute-force method Experiments GPU-based brute-force method: sorting algorithm We have to perform n sorts of m distances Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  10. 10. K-nearest neighbor search GPU-based brute-force method Experiments GPU-based brute-force method: sorting algorithm We have to perform n sorts of m distances Sorting algorithms: Quicksort (recursive algorithm) Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  11. 11. K-nearest neighbor search GPU-based brute-force method Experiments GPU-based brute-force method: sorting algorithm We have to perform n sorts of m distances Sorting algorithms: Quicksort (recursive algorithm) Comb sort Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  12. 12. K-nearest neighbor search GPU-based brute-force method Experiments GPU-based brute-force method: sorting algorithm We have to perform n sorts of m distances Sorting algorithms: Quicksort (recursive algorithm) Comb sort Modified insertion sort Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  13. 13. K-nearest neighbor search GPU-based brute-force method Experiments GPU-based brute-force method: sorting algorithm We have to perform n sorts of m distances Sorting algorithms: Quicksort (recursive algorithm) Comb sort Modified insertion sort Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  14. 14. K-nearest neighbor search GPU-based brute-force method Experiments Experiments Goal Decrease the computation time of the kNN search Compare our GPU-based implementation with a similar CPU-based approach Compare our GPU-based implementation with fast CPU-based approach Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  15. 15. K-nearest neighbor search GPU-based brute-force method Experiments Experiments Goal Decrease the computation time of the kNN search Compare our GPU-based implementation with a similar CPU-based approach Compare our GPU-based implementation with fast CPU-based approach Setup Pentium 4 @ 3.4 GHz, 2GB of DDR2 PC2-5300 memory NVIDIA GeForce 8800 GTX, 768 MB of DDR3 memory, 128 graphic processors @ 1350 MHz Last graphic cards: NVIDIA GeForce GTX 285, 1024 MB of DDR3 memory, 240 graphic processors @ 1476 MHz Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  16. 16. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Synthetic data Parameters n: number of reference points and query points d: dimension of points k: number of neighbors Data: points randomly drawn from U(0, 1). Methods BF-Matlab: Brute-force method implemented in Matlab BF-C: Brute-force method implemented in C BF-CUDA: Brute-force method implemented in CUDA ANN-C++: ANN method implemented in C++ Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  17. 17. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Synthetic data d=8 d=64 d=96 Methods BF-Matlab BF-C ANN-C++ BF-CUDA BF-Matlab BF-C ANN-C++ BF-CUDA BF-Matlab BF-C ANN-C++ BF-CUDA n=1200 0.51 0.13 0.13 0.01 2.24 1.71 0.78 0.02 3.30 2.54 1.20 0.02 n=2400 1.69 0.49 0.33 0.02 9.37 7.28 3.56 0.04 13.89 10.56 4.96 0.05 n=4800 7.84 1.90 0.81 0.04 38.16 26.11 14.66 0.11 55.77 39.26 19.68 0.15 n=9600 35.08 7.53 2.43 0.13 149.76 111.91 59.28 0.40 231.69 168.58 82.45 0.57 Maximum speed-up of BF-CUDA 407 X faster than BF-Matlab 295 X faster than BF-C 148 X faster than ANN-C++ Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  18. 18. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Synthetic data Influence of the parameter k d = 32, n = 4800 Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  19. 19. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Synthetic data Influence of the parameter d k = 20, n = 4800 Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  20. 20. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Synthetic data Influence of the parameter n k = 20, d = 32 Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  21. 21. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Finding similar patches in images Goal Define manually the first patch Find the k most similar patches Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  22. 22. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Finding similar patches in images Goal Define manually the first patch Find the k most similar patches A color patch of size 21 × 21 = point in R1323 Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  23. 23. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Finding similar patches in images Goal Define manually the first patch Find the k most similar patches A color patch of size 21 × 21 = point in R1323 For a color image of size 128 × 128, the problem is equivalent to find the kNN among 16383 points. Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  24. 24. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Finding similar patches in images Data Image size: 128 × 128 = 16384 pixels Patch size: 21 × 21 k = 10 Gray level image (dimension=441) ANN-C++: 3550 ms BF-CUDA: 60 ms Speed-up = 60X Color image (dimension=1323) ANN-C++: 11.03 s BF-CUDA: 0.15 s Speed-up = 75X Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  25. 25. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Texture synthesis Texture synthesis Efros and Leung, ICIP’99 Source Image Is Target Image It Scanline s 2s + 1 L-shape window Principle Fill the target image by random-colored pixels For a given pixel, find the closest L-shape in the source image Assign the value of the L-shape center to the current pixel The L-shape corresponding to a square window of size (2s + 1) × (2s + 1) contains 2s 2 + 2s pixels Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  26. 26. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Texture synthesis Efros and Leung Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  27. 27. K-nearest neighbor search GPU-based brute-force method Experiments Experiments: Texture synthesis Data Source image: 64 × 64 = 4096 pixels Target image: 128 × 128 = 16384 pixels Gray level image (dimension=220) ANN-C++: 720 ms BF-CUDA: 18 ms Speed-up = 40X Color image (dimension=660) ANN-C++: 2 s BF-CUDA: 40 ms Speed-up = 50X Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  28. 28. K-nearest neighbor search GPU-based brute-force method Experiments Conclusion We proposed a GPU-based brute-force kNN search implementation Using GPU, a simple but highly parallelizable method can be faster than a smart algorithm Speed-up obtained on synthetic data up to 400 X faster than a Matlab implementation up to 300 X faster than a C implementation up to 150 X faster than the ANN C++ library Speed-up obtained on real image processing applications Finding similar patches: 75 X Texture synthesis: 50 X Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method
  29. 29. K-nearest neighbor search GPU-based brute-force method Experiments Questions Thanks for your attention Garcia, Nielsen ´ Ecole Polytechnique, Palaiseau, France Searching high-dimensional neighbors: CPU-based tailored data-structures versus GPU-based brute-force method

×