Upcoming SlideShare
×

# 16 17 bag_words

209
-1

Published on

Lecture 8 & 9 for Computer Vision by Dr. Usman Ghani

Published in: Technology, Education
0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total Views
209
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
19
0
Likes
0
Embeds 0
No embeds

No notes for slide
• For that we will use a method called probabilistic latent semantic analysis. pLSA can be thought of as a matrix decomposition. Here is our term-document matrix (documents, words) and we want to find topic vectors common to all documents and mixture coefficients P(z|d) specific to each document. Note that these are all probabilites which sum to one. So a column here is here expressed as a convex combination of the topic vectors. What we would like to see is that the topics correspond to objects. So an image will be expressed as a mixture of different objects and backgrounds.
• For that we will use a method called probabilistic latent semantic analysis. pLSA can be thought of as a matrix decomposition. Here is our term-document matrix (documents, words) and we want to find topic vectors common to all documents and mixture coefficients P(z|d) specific to each document. Note that these are all probabilites which sum to one. So a column here is here expressed as a convex combination of the topic vectors. What we would like to see is that the topics correspond to objects. So an image will be expressed as a mixture of different objects and backgrounds.
• We use maximum likelihood approach to fit parameters of the model. We write down the likelihood of the whole collection and maximize it using EM. M,N goes over all visual words and all documents. P(w|d) factorizes as on previous slide the entries in those matrices are our parameters. n(w,d) are the observed counts in our data
• We use maximum likelihood approach to fit parameters of the model. We write down the likelihood of the whole collection and maximize it using EM. M,N goes over all visual words and all documents. P(w|d) factorizes as on previous slide the entries in those matrices are our parameters. n(w,d) are the observed counts in our data
• We use maximum likelihood approach to fit parameters of the model. We write down the likelihood of the whole collection and maximize it using EM. M,N goes over all visual words and all documents. P(w|d) factorizes as on previous slide the entries in those matrices are our parameters. n(w,d) are the observed counts in our data
• ### 16 17 bag_words

1. 1. Bag-of-Words models Computer Vision Lecture 8Slides from: S. Lazebnik, A. Torralba, L. Fei-Fei, D. Lowe, C. Szurka
2. 2. Bag-of-features models
3. 3. Overview: Bag-of-features models• Origins and motivation• Image representation• Discriminative methods – Nearest-neighbor classification – Support vector machines• Generative methods – Naïve Bayes – Probabilistic Latent Semantic Analysis• Extensions: incorporating spatial information
4. 4. Origin 1: Texture recognition• Texture is characterized by the repetition of basic elements or textons• For stochastic textures, it is the identity of the textons, not their spatial arrangement, that mattersJulesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003
5. 5. Origin 1: Texture recognition histogram Universal texton dictionaryJulesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003
6. 6. Origin 2: Bag-of-words models• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)
7. 7. Origin 2: Bag-of-words models• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/
8. 8. Origin 2: Bag-of-words models• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/
9. 9. Origin 2: Bag-of-words models• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/
10. 10. Bags of features for image classification1. Extract features
11. 11. Bags of features for image classification1. Extract features2. Learn “visual vocabulary”
12. 12. Bags of features for image classification1. Extract features2. Learn “visual vocabulary”3. Quantize features using visual vocabulary
13. 13. Bags of features for image classification1. Extract features2. Learn “visual vocabulary”3. Quantize features using visual vocabulary4. Represent images by frequencies of “visual words”
14. 14. 1. Feature extraction• Regular grid – Vogel & Schiele, 2003 – Fei-Fei & Perona, 2005
15. 15. 1. Feature extraction• Regular grid – Vogel & Schiele, 2003 – Fei-Fei & Perona, 2005• Interest point detector – Csurka et al. 2004 – Fei-Fei & Perona, 2005 – Sivic et al. 2005
16. 16. 1. Feature extraction• Regular grid – Vogel & Schiele, 2003 – Fei-Fei & Perona, 2005• Interest point detector – Csurka et al. 2004 – Fei-Fei & Perona, 2005 – Sivic et al. 2005• Other methods – Random sampling (Vidal-Naquet & Ullman, 2002) – Segmentation-based patches (Barnard et al. 2003)
17. 17. 1. Feature extractionCompute SIFT descriptor Normalize patch [Lowe’99] Detect patches [Mikojaczyk and Schmid ’02] [Mata, Chum, Urban & Pajdla, ’02] [Sivic & Zisserman, ’03] Slide credit: Josef Sivic
18. 18. 1. Feature extraction …
19. 19. 2. Learning the visual vocabulary …
20. 20. 2. Learning the visual vocabulary … Clustering Slide credit: Josef Sivic
21. 21. 2. Learning the visual vocabulary Visual vocabulary … Clustering Slide credit: Josef Sivic
22. 22. K-means clustering• Want to minimize sum of squared Euclidean distances between points xi and their nearest cluster centers mk D( X , M ) = ∑ ∑ ( xi − mk ) 2 cluster k point i in cluster k• Algorithm:• Randomly initialize K cluster centers• Iterate until convergence: – Assign each data point to the nearest center – Recompute each cluster center as the mean of all points assigned to it
23. 23. From clustering to vector quantization• Clustering is a common method for learning a visual vocabulary or codebook – Unsupervised learning process – Each cluster center produced by k-means becomes a codevector – Codebook can be learned on separate training set – Provided the training set is sufficiently representative, the codebook will be “universal”• The codebook is used for quantizing features – A vector quantizer takes a feature vector and maps it to the index of the nearest codevector in a codebook – Codebook = visual vocabulary – Codevector = visual word
24. 24. Example visual vocabulary Fei-Fei et al. 2005
25. 25. Image patch examples of visual words Sivic et al. 2005
26. 26. Visual vocabularies: Issues• How to choose vocabulary size? – Too small: visual words not representative of all patches – Too large: quantization artifacts, overfitting• Generative or discriminative learning?• Computational efficiency – Vocabulary trees (Nister & Stewenius, 2006)
27. 27. 3. Image representationfrequency ….. codewords
28. 28. Image classification• Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?
29. 29. Discriminative and generative methods for bags of features Zebra Non-zebra
30. 30. Discriminative methods• Learn a decision rule (classifier) assigning bag- of-features representations of images to different classes Decision Zebra boundary Non-zebra
31. 31. Classification• Assign input vector to one of two or more classes• Any decision rule divides input space into decision regions separated by decision boundaries
32. 32. Nearest Neighbor Classifier• Assign label of nearest training data point to each test data point from Duda et al. Voronoi partitioning of feature space for two-category 2D and 3D data Source: D. Lowe
33. 33. K-Nearest Neighbors• For a new point, find the k closest points from training data• Labels of the k points “vote” to classify• Works well provided there is lots of data and the distance function is good k=5 Source: D. Lowe
34. 34. Functions for comparing histograms• L1 distance N D(h1 , h2 ) = ∑ | h1 (i ) − h2 (i ) | i =1 D(h1 , h2 ) = ∑ N ( h1 (i) − h2 (i) ) 2• χ2 distance i =1 h1 (i ) + h2 (i )
35. 35. Linear classifiers• Find linear function (hyperplane) to separate positive and negative examples x i positive : xi ⋅ w + b ≥ 0 x i negative : xi ⋅ w + b < 0 Which hyperplane is best? Slide: S. Lazebnik
36. 36. Support vector machines • Find hyperplane that maximizes the margin between the positive and negative examplesC. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining andKnowledge Discovery, 1998
37. 37. Support vector machines • Find hyperplane that maximizes the margin between the positive and negative examples x i positive ( yi = 1) : xi ⋅ w + b ≥ 1 x i negative ( yi = −1) : x i ⋅ w + b ≤ −1 For support vectors, x i ⋅ w + b = ±1 Distance between point | xi ⋅ w + b | and hyperplane: || w || Therefore, the margin is 2 / ||w|| Support vectors MarginC. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining andKnowledge Discovery, 1998
38. 38. Finding the maximum margin hyperplane 1. Maximize margin 2/||w|| 2. Correctly classify all training data: x i positive ( yi = 1) : xi ⋅ w + b ≥ 1 x i negative ( yi = −1) : x i ⋅ w + b ≤ −1C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining andKnowledge Discovery, 1998
39. 39. Nonlinear SVMs• Datasets that are linearly separable work out great: 0 x• But what if the dataset is just too hard? 0 x• We can map it to a higher-dimensional space: x2 0 x Slide credit: Andrew Moore
40. 40. Nonlinear SVMs• General idea: the original input space can always be mapped to some higher- dimensional feature space where the training set is separable: Φ: x → φ(x) Slide credit: Andrew Moore
41. 41. Nonlinear SVMs • The kernel trick: instead of explicitly computing the lifting transformation φ(x), define a kernel function K such that K(xi, xj) = φ(xi ) · φ(xj) • (to be valid, the kernel function must satisfy Mercer’s condition) • This gives a nonlinear decision boundary in the original feature space: ∑ α y K ( x , x) + b i i i iC. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining andKnowledge Discovery, 1998
42. 42. Kernels for bags of features • Histogram intersection kernel: N I (h1 , h2 ) = ∑ min(h1 (i ), h2 (i )) i =1 • Generalized Gaussian kernel:  1 2 K (h1 , h2 ) = exp − D(h1 , h2 )   A  • D can be Euclidean distance, χ2 distanceJ. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid, Local Features and Kernels for Classifcationof Texture and Object Categories: A Comprehensive Study, IJCV 2007
43. 43. What about multi-class SVMs?• Unfortunately, there is no “definitive” multi-class SVM formulation• In practice, we have to obtain a multi-class SVM by combining multiple two-class SVMs• One vs. others – Traning: learn an SVM for each class vs. the others – Testing: apply each SVM to test example and assign to it the class of the SVM that returns the highest decision value• One vs. one – Training: learn an SVM for each pair of classes – Testing: each learned SVM “votes” for a class to assign to the test example Slide: S. Lazebnik
44. 44. SVMs: Pros and cons• Pros – Many publicly available SVM packages: http://www.kernel-machines.org/software – Kernel-based framework is very powerful, flexible – SVMs work very well in practice, even with very small training sample sizes• Cons – No “direct” multi-class SVM, must combine two- class SVMs – Computation, memory • During training time, must compute matrix of kernel values for every pair of examples • Learning can take a very long time for large-scale problems Slide: S. Lazebnik
45. 45. Summary: Discriminative methods• Nearest-neighbor and k-nearest-neighbor classifiers – L1 distance, χ2 distance, quadratic distance, – Support vector machines – Linear classifiers – Margin maximization – The kernel trick – Kernel functions: histogram intersection, generalized Gaussian, pyramid match – Multi-class• Of course, there are many other classifiers out there – Neural networks, boosting, decision trees, …
46. 46. Generative learning methods for bags of features• Model the probability of a bag of features given a class
47. 47. Generative methods• We will cover two models, both inspired by text document analysis: – Naïve Bayes – Probabilistic Latent Semantic Analysis
48. 48. The Naïve Bayes model• Assume that each feature is conditionally independent given the class N p ( f1 ,  , f N | c) = ∏ p ( f i | c) i =1 fi: ith feature in the image N: number of features in the image Csurka et al. 2004
49. 49. The Naïve Bayes model• Assume that each feature is conditionally independent given the class N M p( f1 ,  , f N | c) = ∏ p ( f i | c) = ∏ p ( w j | c) n(w j ) i =1 j =1 fi: ith feature in the image N: number of features in the image wj: jth visual word in the vocabulary M: size of visual vocabulary n(wj): number of features of type wj in the image Csurka et al. 2004
50. 50. The Naïve Bayes model• Assume that each feature is conditionally independent given the class N M p( f1 ,  , f N | c) = ∏ p ( f i | c) = ∏ p ( w j | c) n(w j ) i =1 j =1 No. of features of type wj in training images of class cp(wj | c) = Total no. of features in training images of class c Csurka et al. 2004
51. 51. The Naïve Bayes model• Maximum A Posteriori decision: M c* = arg max c p (c)∏ p ( w j | c) n(w j ) j =1 M = arg max c log p (c) + ∑ n( w j ) log p ( w j | c) j =1 (you should compute the log of the likelihood instead of the likelihood itself in order to avoid underflow) Csurka et al. 2004
52. 52. The Naïve Bayes model• “Graphical model”: c w N Csurka et al. 2004
53. 53. Probabilistic Latent Semantic Analysis = p1 + p2 + p3Image zebra grass tree “visual topics” T. Hofmann, Probabilistic Latent Semantic Analysis, UAI 1999
54. 54. Probabilistic Latent Semantic Analysis• Unsupervised technique• Two-level generative model: a document is a mixture of topics, and each topic has its own characteristic word distribution d z w document topic word P(z|d) P(w|z) T. Hofmann, Probabilistic Latent Semantic Analysis, UAI 1999
55. 55. Probabilistic Latent Semantic Analysis• Unsupervised technique• Two-level generative model: a document is a mixture of topics, and each topic has its own characteristic word distribution d z w K p ( wi | d j ) = ∑ p ( wi | z k ) p ( z k | d j ) k =1 T. Hofmann, Probabilistic Latent Semantic Analysis, UAI 1999
56. 56. The pLSA model K p ( wi | d j ) = ∑ p ( wi | z k ) p ( z k | d j ) k =1Probability of word i Probability of Probability of in document j word i given topic k given (known) topic k document j (unknown) (unknown)
57. 57. The pLSA model K p ( wi | d j ) = ∑ p ( wi | z k ) p ( z k | d j ) k =1 documents topics documentswords words topics p(zk|dj) p(wi|dj) = p(wi|zk) Observed codeword Codeword distributions Class distributions distributions per topic (class) per image (M×N) (M×K) (K×N)
58. 58. Learning pLSA parametersMaximize likelihood of data: Observed counts of word i in document j M … number of codewords N … number of images Slide credit: Josef Sivic
59. 59. Inference• Finding the most likely topic (class) for an image: ∗ z = arg max p ( z | d ) z
60. 60. Inference • Finding the most likely topic (class) for an image: ∗ z = arg max p ( z | d ) z• Finding the most likely topic (class) for a visual word in a given image: z ∗ = arg max p ( z | w, d ) z
61. 61. Topic discovery in imagesJ. Sivic, B. Russell, A. Efros, A. Zisserman, B. Freeman, Discovering Objects and their Locationin Images, ICCV 2005
62. 62. Application of pLSA: Action recognition Space-time interest pointsJuan Carlos Niebles, Hongcheng Wang and Li Fei-Fei, Unsupervised Learning of Human ActionCategories Using Spatial-Temporal Words, IJCV 2008.
63. 63. Application of pLSA: Action recognitionJuan Carlos Niebles, Hongcheng Wang and Li Fei-Fei, Unsupervised Learning of Human ActionCategories Using Spatial-Temporal Words, IJCV 2008.
64. 64. pLSA model K p ( wi | d j ) = ∑ p ( wi | z k ) p ( z k | d j ) k =1 Probability of word i Probability of Probability of in video j word i given topic k given (known) topic k video j (unknown) (unknown)– wi = spatial-temporal word– dj = video– n(wi, dj) = co-occurrence table (# of occurrences of word wi in video dj)– z = topic, corresponding to an action
65. 65. Action recognition example
66. 66. Multiple Actions
67. 67. Multiple Actions
68. 68. Summary: Generative models• Naïve Bayes – Unigram models in document analysis – Assumes conditional independence of words given class – Parameter estimation: frequency counting• Probabilistic Latent Semantic Analysis – Unsupervised technique – Each document is a mixture of topics (image is a mixture of classes) – Can be thought of as matrix decomposition
1. #### A particular slide catching your eye?

Clipping is a handy way to collect important slides you want to go back to later.