Visual Search


Published on

A gentle introduction to some fundamentals of image retrieval

Published in: Technology

Visual Search

  1. 1. Visual Search A behind-the-scenes look at image retrieval Amit Prabhudesai SAIT-India
  2. 2. Outline <ul><li>What is Visual search? </li></ul><ul><li>Use-cases & applications </li></ul><ul><li>Basics of a Image Retrieval system </li></ul><ul><ul><li>Descriptors </li></ul></ul><ul><ul><li>Similarity measures </li></ul></ul><ul><ul><li>Indexing schemes </li></ul></ul><ul><li>How do you measure performance? </li></ul>
  3. 3. Why do we need visual search? How do I find what I’m looking for?!
  4. 4. What is visual search? Text query, textual results Text query, visual results Visual query, visual results … a.k.a. Visual Search
  5. 5. Visual search – use-cases <ul><li>Search by browsing </li></ul><ul><ul><li>User begins by submitting a keyword for the object-of-interest </li></ul></ul><ul><ul><li>System returns visual results (images/videos) </li></ul></ul><ul><ul><li>User browses through them and marks interesting results and asks system to return similar content </li></ul></ul>Search by browsing … a.k.a. “show me similar content”
  6. 6. Uses-cases of visual search <ul><li>Search by example </li></ul><ul><ul><li>User has a specific query – e.g. she may be looking for red-cars </li></ul></ul><ul><ul><li>Uploads an example of the object-of-interest </li></ul></ul><ul><ul><li>System returns similar visual content </li></ul></ul>Search by example … a.k.a. “Show me all red-cars!”
  7. 7. Uses-cases (contd.) … <ul><li>Search by drawing </li></ul><ul><ul><li>“ A picture speaks better than a thousand words”!! </li></ul></ul><ul><ul><li>User draws out an object/concept that she has in mind </li></ul></ul><ul><ul><li>System returns visual content similar to the object drawn </li></ul></ul>Search by drawing …
  8. 8. Use-cases (contd.) … <ul><li>Search by category </li></ul><ul><ul><li>User wants to retrieve all visual content in a particular category </li></ul></ul><ul><ul><li>Difficult problem! </li></ul></ul><ul><ul><li>Semantic gap : gap between the user’s understanding and the features computed by a machine </li></ul></ul>Search by category …
  9. 9. Applications of visual search <ul><li>Art galleries & museum management </li></ul><ul><li>Searching product catalogs </li></ul><ul><li>Architectural & engineering design </li></ul><ul><li>Geographical information systems </li></ul><ul><li>Picture archiving </li></ul><ul><li>Law-enforcement & criminal investigations </li></ul>
  10. 10. Visual search a.k.a. Content Based Image Retrieval (CBIR) Block-diagram of a typical content-based image retrieval system Comparison Query Index Query Index Result Image database
  11. 11. Components of a CBIR system <ul><li>Image content descriptor </li></ul><ul><ul><li>Compute machine-understandable attributes </li></ul></ul><ul><li>Similarity/distance metrics </li></ul><ul><ul><li>Measure the similarity (or lack thereof) between query and database sample </li></ul></ul><ul><li>Indexing schemes </li></ul><ul><ul><li>How do you efficiently search the image/visual content database </li></ul></ul><ul><li>Relevance feedback </li></ul><ul><ul><li>Use the users’ choices to improve retrieval </li></ul></ul><ul><li>Performance measurement </li></ul><ul><ul><li>Metrics to measure effectiveness </li></ul></ul>
  12. 12. Image content descriptors <ul><li>Different attributes are used </li></ul><ul><ul><li>Color </li></ul></ul><ul><ul><li>Shape </li></ul></ul><ul><ul><li>Texture </li></ul></ul><ul><ul><li>Spatial layout </li></ul></ul>
  13. 13. Image content descriptors – Color <ul><li>Used extensively for image retrieval </li></ul><ul><ul><li>Motivation: human visual system </li></ul></ul><ul><ul><li>Simple & intuitive! </li></ul></ul><ul><ul><li>Not very discriminative </li></ul></ul><ul><ul><li>Used as a first pass to filter out unlikely examples </li></ul></ul>
  14. 14. Image descriptors – Color Apples are red … … But tomatoes are too!!!
  15. 15. Image descriptors – Color <ul><li>Color descriptors </li></ul><ul><ul><li>Color histograms – local/global </li></ul></ul><ul><ul><li>Color moments </li></ul></ul><ul><ul><li>Color coherence vector </li></ul></ul><ul><ul><li>Color correlogram </li></ul></ul>
  16. 16. Image descriptors - shape <ul><li>Segment foreground ‘objects’ </li></ul><ul><li>Shape can be used to describe these objects </li></ul><ul><li>Desirable attributes </li></ul><ul><ul><li>Should be invariant to translation, rotation and scaling </li></ul></ul>
  17. 17. Image descriptors – shape <ul><li>Classical shape representation uses moment variants </li></ul><ul><li>Boundary based methods </li></ul><ul><ul><li>Turning function or Turning angle </li></ul></ul><ul><li>Geometrical attributes </li></ul><ul><ul><li>Aspect ratios, (relative) dimensions </li></ul></ul>
  18. 18. Image descriptors – Texture <ul><li>Different scenes may have same color! </li></ul><ul><li>Taking a cue from the human visual system (HVS) </li></ul>
  19. 19. Image descriptors – Texture Texture differentiates between a Lawn and a Forest
  20. 20. Image descriptors – Texture <ul><li>Wavelet transform features </li></ul><ul><ul><li>Multi-resolution approach to texture analysis </li></ul></ul><ul><ul><li>Texture described at various scales </li></ul></ul><ul><li>Gabor filter features </li></ul><ul><ul><li>Orientation and scale-tunable line (bar) detector </li></ul></ul><ul><li>Tamura features & Wold features </li></ul><ul><ul><li>Based on characteristics like coarseness, contrast, directionality, regularity (or lack thereof) </li></ul></ul>
  21. 21. Image descriptors – spatial information <ul><li>Sky is blue … but so is water! </li></ul><ul><li>What differentiates them is spatial layout! </li></ul><ul><li>Some common descriptors </li></ul><ul><ul><li>2D strings </li></ul></ul><ul><ul><li>Spatial quad-tree </li></ul></ul><ul><ul><li>Symbolic image </li></ul></ul>
  22. 22. The whole is greater than the sum of the parts!! <ul><li>Any one simple descriptor cannot give results required in ‘usable’ systems! </li></ul><ul><li>State-of-the-art systems use combination of descriptors </li></ul>
  23. 23. Similarity/distance measures <ul><li>Exact match cannot be found! </li></ul><ul><li>Similarity/distance measured by </li></ul><ul><ul><li>Quadratic-form distance </li></ul></ul><ul><ul><li>Mahalanobis distance </li></ul></ul><ul><ul><li>Minkowski-form distance </li></ul></ul><ul><ul><li>Histogram intersection </li></ul></ul><ul><ul><li>Kullback-Leibler divergence </li></ul></ul><ul><li>Target application decides which distance measure is used </li></ul>
  24. 24. Indexing scheme <ul><li>Features typically have high dimensionality </li></ul><ul><li>Efficient indexing becomes a critical performance issue </li></ul><ul><ul><li>Dimensionality reduction (e.g., PCA) </li></ul></ul><ul><ul><li>R-tree, linear quad-trees, K-d-B-Tree, grid files </li></ul></ul>
  25. 25. Performance Evaluation <ul><li>Precision </li></ul><ul><ul><li>Precision is the fraction of retrieved images that are indeed relevant to the query </li></ul></ul><ul><li>Recall </li></ul><ul><ul><li>Recall is the fraction of relevant images returned by the query </li></ul></ul><ul><li>Trade-off between precision and recall </li></ul><ul><ul><li>Recall tends to increase as # retrieved items increases; precision decreases </li></ul></ul>
  26. 26. What is out there? <ul><li>Some Web Resources </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li> </li></ul></ul><ul><ul><li>http:// /images </li></ul></ul>
  27. 27. Thank You