Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
WhittleSearch: Image Search with Relative
Attribute Feedback
CVPR 2012
Adriana Kovashka, Devi Parikh and Kristen Grauman
P...
Keywords:
Not enough to describe an image.
Content-based:
Limited by the "semantic gap"
Relative attributes
Shoe: Higher h...
Background
Binary Relevance Feedback
● User marks some images as relevant or irrelevant given
their search target.
● We le...
Binary Relevance Feedback
Example
Proposed Approach
Relative Attributes Feedback
Relative Attributes Feedback
Idea: Refine image search by user feedbacks
Relative Attribute Feedback
Example
Relative Attributes Feedback
Learning to predict relative attributes
● Mechanical Turk
Interface for image-level relative
attribute annotations
Ranking functions for M attributes
Joachims. Originally for ranking web pages based
on click data.
X: Feature vector
SVMRank
Thorsten Joachims, "Optimizing Search Engines Using Clickthrough Data", KDD 2002
Updating the scoring function from
feedback
● Case 1:
● Case 2:
● Case 3:
Updating the scoring function from
feedback
● Naive method
o Update the scores by counting satisfactions of attribute cons...
Updating the scoring function from
feedback
● Advanced method
o Hybrid feedback approach
o Update the scores by re-trainin...
Updating the scoring function from
feedback
Experiments
● Datasets
o Shoes: 14,658 shoe images from like.com with 10
relative attributes (‘open’, ‘shiny’, ‘formal’, ‘...
Experiments
● Impact of iterative feedback
Experiments
● Impact of amount of feedback
● Impact of reference images
o Impact of the types of reference images available for
feedback.
Experiments
Experiments
● Ranking accuracy with human-given
feedback
o 16 queries per dataset after one round of 8
feedback statements.
Experiments
● Consistency of Relative Supervision Types
o Class-level vs. instance-level
 Instance-level supervision outp...
Questions?
● Sorting of the relative attribute feedbacks.
o How do we model the ranking of relative attribute?
o For examp...
Upcoming SlideShare
Loading in …5
×

WhittleSearch: Image Search with Relative Attribute Feedback

132 views

Published on

In this presentation, we reviewed WhittleSearch: Image Search with Relative Attribute Feedback by Adriana Kovashka, Devi Parikh and Kristen Grauman in CVPR 2012.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

WhittleSearch: Image Search with Relative Attribute Feedback

  1. 1. WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka, Devi Parikh and Kristen Grauman Presenter: Xiyang Dai, Ruofei Du 10/05/2014 University of Texas at Austin Toyota Technological Institute Chicago (TTIC)
  2. 2. Keywords: Not enough to describe an image. Content-based: Limited by the "semantic gap" Relative attributes Shoe: Higher heels Man: Younger or older Introduction Relative attribute feedback for Image Retrieval
  3. 3. Background Binary Relevance Feedback ● User marks some images as relevant or irrelevant given their search target. ● We learn a binary classifier using the relevant images as positives and the irrelevant images as negatives, and rank images in the dataset by the classifier outputs.
  4. 4. Binary Relevance Feedback Example
  5. 5. Proposed Approach Relative Attributes Feedback
  6. 6. Relative Attributes Feedback Idea: Refine image search by user feedbacks
  7. 7. Relative Attribute Feedback Example
  8. 8. Relative Attributes Feedback Learning to predict relative attributes ● Mechanical Turk
  9. 9. Interface for image-level relative attribute annotations
  10. 10. Ranking functions for M attributes Joachims. Originally for ranking web pages based on click data. X: Feature vector
  11. 11. SVMRank Thorsten Joachims, "Optimizing Search Engines Using Clickthrough Data", KDD 2002
  12. 12. Updating the scoring function from feedback ● Case 1: ● Case 2: ● Case 3:
  13. 13. Updating the scoring function from feedback ● Naive method o Update the scores by counting satisfactions of attribute constraints
  14. 14. Updating the scoring function from feedback ● Advanced method o Hybrid feedback approach o Update the scores by re-training ranking functions
  15. 15. Updating the scoring function from feedback
  16. 16. Experiments ● Datasets o Shoes: 14,658 shoe images from like.com with 10 relative attributes (‘open’, ‘shiny’, ‘formal’, ‘sporty’, etc.) o PubFig: 772 images from 8 people and 11 attributes (‘young’, ‘round face’, etc.) o OSR: 2,688 images from 8 categories and 6 attributes (‘openness’, ‘perspective’, etc.)
  17. 17. Experiments ● Impact of iterative feedback
  18. 18. Experiments ● Impact of amount of feedback
  19. 19. ● Impact of reference images o Impact of the types of reference images available for feedback. Experiments
  20. 20. Experiments ● Ranking accuracy with human-given feedback o 16 queries per dataset after one round of 8 feedback statements.
  21. 21. Experiments ● Consistency of Relative Supervision Types o Class-level vs. instance-level  Instance-level supervision outperforms class-level supervision. o Absolute vs. relative  Disagreement using relative responses is lower than absolute responses.
  22. 22. Questions? ● Sorting of the relative attribute feedbacks. o How do we model the ranking of relative attribute? o For example, which is more important for image retrieval: “more natural” or “more open” for a scene? ● Could category attributes be applied to the relative attribute feedbacks? o For example, “more thinner” or “higher heels” might refer to a completely different class to describe a shoe: “boots” or “high-heel” o Shall we sort by relative attribute in the results?

×