• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
16 17 bag_words
 

16 17 bag_words

on

  • 234 views

Lecture 8 & 9 for Computer Vision by Dr. Usman Ghani

Lecture 8 & 9 for Computer Vision by Dr. Usman Ghani

Statistics

Views

Total Views
234
Views on SlideShare
234
Embed Views
0

Actions

Likes
0
Downloads
16
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • For that we will use a method called probabilistic latent semantic analysis. pLSA can be thought of as a matrix decomposition. Here is our term-document matrix (documents, words) and we want to find topic vectors common to all documents and mixture coefficients P(z|d) specific to each document. Note that these are all probabilites which sum to one. So a column here is here expressed as a convex combination of the topic vectors. What we would like to see is that the topics correspond to objects. So an image will be expressed as a mixture of different objects and backgrounds.
  • For that we will use a method called probabilistic latent semantic analysis. pLSA can be thought of as a matrix decomposition. Here is our term-document matrix (documents, words) and we want to find topic vectors common to all documents and mixture coefficients P(z|d) specific to each document. Note that these are all probabilites which sum to one. So a column here is here expressed as a convex combination of the topic vectors. What we would like to see is that the topics correspond to objects. So an image will be expressed as a mixture of different objects and backgrounds.
  • We use maximum likelihood approach to fit parameters of the model. We write down the likelihood of the whole collection and maximize it using EM. M,N goes over all visual words and all documents. P(w|d) factorizes as on previous slide the entries in those matrices are our parameters. n(w,d) are the observed counts in our data
  • We use maximum likelihood approach to fit parameters of the model. We write down the likelihood of the whole collection and maximize it using EM. M,N goes over all visual words and all documents. P(w|d) factorizes as on previous slide the entries in those matrices are our parameters. n(w,d) are the observed counts in our data
  • We use maximum likelihood approach to fit parameters of the model. We write down the likelihood of the whole collection and maximize it using EM. M,N goes over all visual words and all documents. P(w|d) factorizes as on previous slide the entries in those matrices are our parameters. n(w,d) are the observed counts in our data

16 17 bag_words 16 17 bag_words Presentation Transcript

  • Bag-of-Words models Computer Vision Lecture 8Slides from: S. Lazebnik, A. Torralba, L. Fei-Fei, D. Lowe, C. Szurka
  • Bag-of-features models
  • Overview: Bag-of-features models• Origins and motivation• Image representation• Discriminative methods – Nearest-neighbor classification – Support vector machines• Generative methods – Naïve Bayes – Probabilistic Latent Semantic Analysis• Extensions: incorporating spatial information
  • Origin 1: Texture recognition• Texture is characterized by the repetition of basic elements or textons• For stochastic textures, it is the identity of the textons, not their spatial arrangement, that mattersJulesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003
  • Origin 1: Texture recognition histogram Universal texton dictionaryJulesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003
  • Origin 2: Bag-of-words models• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)
  • Origin 2: Bag-of-words models• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/
  • Origin 2: Bag-of-words models• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/
  • Origin 2: Bag-of-words models• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/
  • Bags of features for image classification1. Extract features
  • Bags of features for image classification1. Extract features2. Learn “visual vocabulary”
  • Bags of features for image classification1. Extract features2. Learn “visual vocabulary”3. Quantize features using visual vocabulary
  • Bags of features for image classification1. Extract features2. Learn “visual vocabulary”3. Quantize features using visual vocabulary4. Represent images by frequencies of “visual words”
  • 1. Feature extraction• Regular grid – Vogel & Schiele, 2003 – Fei-Fei & Perona, 2005
  • 1. Feature extraction• Regular grid – Vogel & Schiele, 2003 – Fei-Fei & Perona, 2005• Interest point detector – Csurka et al. 2004 – Fei-Fei & Perona, 2005 – Sivic et al. 2005
  • 1. Feature extraction• Regular grid – Vogel & Schiele, 2003 – Fei-Fei & Perona, 2005• Interest point detector – Csurka et al. 2004 – Fei-Fei & Perona, 2005 – Sivic et al. 2005• Other methods – Random sampling (Vidal-Naquet & Ullman, 2002) – Segmentation-based patches (Barnard et al. 2003)
  • 1. Feature extractionCompute SIFT descriptor Normalize patch [Lowe’99] Detect patches [Mikojaczyk and Schmid ’02] [Mata, Chum, Urban & Pajdla, ’02] [Sivic & Zisserman, ’03] Slide credit: Josef Sivic
  • 1. Feature extraction …
  • 2. Learning the visual vocabulary …
  • 2. Learning the visual vocabulary … Clustering Slide credit: Josef Sivic
  • 2. Learning the visual vocabulary Visual vocabulary … Clustering Slide credit: Josef Sivic
  • K-means clustering• Want to minimize sum of squared Euclidean distances between points xi and their nearest cluster centers mk D( X , M ) = ∑ ∑ ( xi − mk ) 2 cluster k point i in cluster k• Algorithm:• Randomly initialize K cluster centers• Iterate until convergence: – Assign each data point to the nearest center – Recompute each cluster center as the mean of all points assigned to it
  • From clustering to vector quantization• Clustering is a common method for learning a visual vocabulary or codebook – Unsupervised learning process – Each cluster center produced by k-means becomes a codevector – Codebook can be learned on separate training set – Provided the training set is sufficiently representative, the codebook will be “universal”• The codebook is used for quantizing features – A vector quantizer takes a feature vector and maps it to the index of the nearest codevector in a codebook – Codebook = visual vocabulary – Codevector = visual word
  • Example visual vocabulary Fei-Fei et al. 2005
  • Image patch examples of visual words Sivic et al. 2005
  • Visual vocabularies: Issues• How to choose vocabulary size? – Too small: visual words not representative of all patches – Too large: quantization artifacts, overfitting• Generative or discriminative learning?• Computational efficiency – Vocabulary trees (Nister & Stewenius, 2006)
  • 3. Image representationfrequency ….. codewords
  • Image classification• Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?
  • Discriminative and generative methods for bags of features Zebra Non-zebra
  • Discriminative methods• Learn a decision rule (classifier) assigning bag- of-features representations of images to different classes Decision Zebra boundary Non-zebra
  • Classification• Assign input vector to one of two or more classes• Any decision rule divides input space into decision regions separated by decision boundaries
  • Nearest Neighbor Classifier• Assign label of nearest training data point to each test data point from Duda et al. Voronoi partitioning of feature space for two-category 2D and 3D data Source: D. Lowe
  • K-Nearest Neighbors• For a new point, find the k closest points from training data• Labels of the k points “vote” to classify• Works well provided there is lots of data and the distance function is good k=5 Source: D. Lowe
  • Functions for comparing histograms• L1 distance N D(h1 , h2 ) = ∑ | h1 (i ) − h2 (i ) | i =1 D(h1 , h2 ) = ∑ N ( h1 (i) − h2 (i) ) 2• χ2 distance i =1 h1 (i ) + h2 (i )
  • Linear classifiers• Find linear function (hyperplane) to separate positive and negative examples x i positive : xi ⋅ w + b ≥ 0 x i negative : xi ⋅ w + b < 0 Which hyperplane is best? Slide: S. Lazebnik
  • Support vector machines • Find hyperplane that maximizes the margin between the positive and negative examplesC. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining andKnowledge Discovery, 1998
  • Support vector machines • Find hyperplane that maximizes the margin between the positive and negative examples x i positive ( yi = 1) : xi ⋅ w + b ≥ 1 x i negative ( yi = −1) : x i ⋅ w + b ≤ −1 For support vectors, x i ⋅ w + b = ±1 Distance between point | xi ⋅ w + b | and hyperplane: || w || Therefore, the margin is 2 / ||w|| Support vectors MarginC. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining andKnowledge Discovery, 1998
  • Finding the maximum margin hyperplane 1. Maximize margin 2/||w|| 2. Correctly classify all training data: x i positive ( yi = 1) : xi ⋅ w + b ≥ 1 x i negative ( yi = −1) : x i ⋅ w + b ≤ −1C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining andKnowledge Discovery, 1998
  • Nonlinear SVMs• Datasets that are linearly separable work out great: 0 x• But what if the dataset is just too hard? 0 x• We can map it to a higher-dimensional space: x2 0 x Slide credit: Andrew Moore
  • Nonlinear SVMs• General idea: the original input space can always be mapped to some higher- dimensional feature space where the training set is separable: Φ: x → φ(x) Slide credit: Andrew Moore
  • Nonlinear SVMs • The kernel trick: instead of explicitly computing the lifting transformation φ(x), define a kernel function K such that K(xi, xj) = φ(xi ) · φ(xj) • (to be valid, the kernel function must satisfy Mercer’s condition) • This gives a nonlinear decision boundary in the original feature space: ∑ α y K ( x , x) + b i i i iC. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining andKnowledge Discovery, 1998
  • Kernels for bags of features • Histogram intersection kernel: N I (h1 , h2 ) = ∑ min(h1 (i ), h2 (i )) i =1 • Generalized Gaussian kernel:  1 2 K (h1 , h2 ) = exp − D(h1 , h2 )   A  • D can be Euclidean distance, χ2 distanceJ. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid, Local Features and Kernels for Classifcationof Texture and Object Categories: A Comprehensive Study, IJCV 2007
  • What about multi-class SVMs?• Unfortunately, there is no “definitive” multi-class SVM formulation• In practice, we have to obtain a multi-class SVM by combining multiple two-class SVMs• One vs. others – Traning: learn an SVM for each class vs. the others – Testing: apply each SVM to test example and assign to it the class of the SVM that returns the highest decision value• One vs. one – Training: learn an SVM for each pair of classes – Testing: each learned SVM “votes” for a class to assign to the test example Slide: S. Lazebnik
  • SVMs: Pros and cons• Pros – Many publicly available SVM packages: http://www.kernel-machines.org/software – Kernel-based framework is very powerful, flexible – SVMs work very well in practice, even with very small training sample sizes• Cons – No “direct” multi-class SVM, must combine two- class SVMs – Computation, memory • During training time, must compute matrix of kernel values for every pair of examples • Learning can take a very long time for large-scale problems Slide: S. Lazebnik
  • Summary: Discriminative methods• Nearest-neighbor and k-nearest-neighbor classifiers – L1 distance, χ2 distance, quadratic distance, – Support vector machines – Linear classifiers – Margin maximization – The kernel trick – Kernel functions: histogram intersection, generalized Gaussian, pyramid match – Multi-class• Of course, there are many other classifiers out there – Neural networks, boosting, decision trees, …
  • Generative learning methods for bags of features• Model the probability of a bag of features given a class
  • Generative methods• We will cover two models, both inspired by text document analysis: – Naïve Bayes – Probabilistic Latent Semantic Analysis
  • The Naïve Bayes model• Assume that each feature is conditionally independent given the class N p ( f1 ,  , f N | c) = ∏ p ( f i | c) i =1 fi: ith feature in the image N: number of features in the image Csurka et al. 2004
  • The Naïve Bayes model• Assume that each feature is conditionally independent given the class N M p( f1 ,  , f N | c) = ∏ p ( f i | c) = ∏ p ( w j | c) n(w j ) i =1 j =1 fi: ith feature in the image N: number of features in the image wj: jth visual word in the vocabulary M: size of visual vocabulary n(wj): number of features of type wj in the image Csurka et al. 2004
  • The Naïve Bayes model• Assume that each feature is conditionally independent given the class N M p( f1 ,  , f N | c) = ∏ p ( f i | c) = ∏ p ( w j | c) n(w j ) i =1 j =1 No. of features of type wj in training images of class cp(wj | c) = Total no. of features in training images of class c Csurka et al. 2004
  • The Naïve Bayes model• Maximum A Posteriori decision: M c* = arg max c p (c)∏ p ( w j | c) n(w j ) j =1 M = arg max c log p (c) + ∑ n( w j ) log p ( w j | c) j =1 (you should compute the log of the likelihood instead of the likelihood itself in order to avoid underflow) Csurka et al. 2004
  • The Naïve Bayes model• “Graphical model”: c w N Csurka et al. 2004
  • Probabilistic Latent Semantic Analysis = p1 + p2 + p3Image zebra grass tree “visual topics” T. Hofmann, Probabilistic Latent Semantic Analysis, UAI 1999
  • Probabilistic Latent Semantic Analysis• Unsupervised technique• Two-level generative model: a document is a mixture of topics, and each topic has its own characteristic word distribution d z w document topic word P(z|d) P(w|z) T. Hofmann, Probabilistic Latent Semantic Analysis, UAI 1999
  • Probabilistic Latent Semantic Analysis• Unsupervised technique• Two-level generative model: a document is a mixture of topics, and each topic has its own characteristic word distribution d z w K p ( wi | d j ) = ∑ p ( wi | z k ) p ( z k | d j ) k =1 T. Hofmann, Probabilistic Latent Semantic Analysis, UAI 1999
  • The pLSA model K p ( wi | d j ) = ∑ p ( wi | z k ) p ( z k | d j ) k =1Probability of word i Probability of Probability of in document j word i given topic k given (known) topic k document j (unknown) (unknown)
  • The pLSA model K p ( wi | d j ) = ∑ p ( wi | z k ) p ( z k | d j ) k =1 documents topics documentswords words topics p(zk|dj) p(wi|dj) = p(wi|zk) Observed codeword Codeword distributions Class distributions distributions per topic (class) per image (M×N) (M×K) (K×N)
  • Learning pLSA parametersMaximize likelihood of data: Observed counts of word i in document j M … number of codewords N … number of images Slide credit: Josef Sivic
  • Inference• Finding the most likely topic (class) for an image: ∗ z = arg max p ( z | d ) z
  • Inference • Finding the most likely topic (class) for an image: ∗ z = arg max p ( z | d ) z• Finding the most likely topic (class) for a visual word in a given image: z ∗ = arg max p ( z | w, d ) z
  • Topic discovery in imagesJ. Sivic, B. Russell, A. Efros, A. Zisserman, B. Freeman, Discovering Objects and their Locationin Images, ICCV 2005
  • Application of pLSA: Action recognition Space-time interest pointsJuan Carlos Niebles, Hongcheng Wang and Li Fei-Fei, Unsupervised Learning of Human ActionCategories Using Spatial-Temporal Words, IJCV 2008.
  • Application of pLSA: Action recognitionJuan Carlos Niebles, Hongcheng Wang and Li Fei-Fei, Unsupervised Learning of Human ActionCategories Using Spatial-Temporal Words, IJCV 2008.
  • pLSA model K p ( wi | d j ) = ∑ p ( wi | z k ) p ( z k | d j ) k =1 Probability of word i Probability of Probability of in video j word i given topic k given (known) topic k video j (unknown) (unknown)– wi = spatial-temporal word– dj = video– n(wi, dj) = co-occurrence table (# of occurrences of word wi in video dj)– z = topic, corresponding to an action
  • Action recognition example
  • Multiple Actions
  • Multiple Actions
  • Summary: Generative models• Naïve Bayes – Unigram models in document analysis – Assumes conditional independence of words given class – Parameter estimation: frequency counting• Probabilistic Latent Semantic Analysis – Unsupervised technique – Each document is a mixture of topics (image is a mixture of classes) – Can be thought of as matrix decomposition