Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Visual Search


Published on

Look at basics of image retrieval, some existing solutions and technology and challenges.

Published in: Technology
  • Be the first to comment

Visual Search

  1. 1. Visual Search aka Content-based Image Retrieval Amit Prabhudesai, SISO
  2. 2. Outline <ul><li>What is Visual search? </li></ul><ul><ul><li>Why do we need it? </li></ul></ul><ul><li>Use-cases & applications </li></ul><ul><li>Some existing solutions/technology </li></ul><ul><li>Basics of an Image Retrieval system </li></ul><ul><ul><li>Descriptors </li></ul></ul><ul><ul><li>Similarity measures </li></ul></ul><ul><ul><li>Indexing schemes </li></ul></ul><ul><li>How do you measure performance? </li></ul><ul><ul><li>How do existing solutions/technology fare? </li></ul></ul>
  3. 3. Why do we need visual search? How do I find what I’m looking for?!
  4. 4. Some telling numbers … <ul><li>Number of photos uploaded as of May2009 – 15 billion </li></ul><ul><li>220 million new photos per week </li></ul><ul><li>550,000 images served per second </li></ul><ul><li>is now the biggest photo-sharing site in the world! </li></ul>Source:
  5. 5. Searching Images is like …
  6. 6. What is visual search? Text query, textual results Text query, visual results Visual query, visual results … a.k.a. Visual Search
  7. 7. A Picture speaks better than … Visual search – why type when you can see it :
  8. 8. Visual search – specifying queries
  9. 9. Visual search – specifying queries
  10. 10. Visual search – specifying queries Snap a picture of something you like & upload it!
  11. 11. Visual search – use-cases <ul><li>Search by browsing </li></ul><ul><ul><li>User begins by submitting a keyword for the object-of-interest/uploading an image </li></ul></ul><ul><ul><li>System returns visual results (images/videos) </li></ul></ul><ul><ul><li>User browses through them and marks interesting results and asks system to return similar content </li></ul></ul>Search by browsing … a.k.a. “show me similar content”
  12. 12. Visual search – use-cases <ul><li>Search by drawing </li></ul><ul><ul><li>“ A picture speaks better than a thousand words”!! </li></ul></ul><ul><ul><li>User draws out an object/concept that she has in mind </li></ul></ul><ul><ul><li>System returns visual content similar to the object drawn </li></ul></ul>Search by drawing … Searching for LOVE ? ;)
  13. 13. Uses-cases … <ul><li>Search by example </li></ul><ul><ul><li>User has a specific query – e.g. she may be looking for red-cars </li></ul></ul><ul><ul><li>Uploads an example of the object-of-interest </li></ul></ul><ul><ul><li>System returns similar visual content </li></ul></ul>Search by example … a.k.a. “Show me all red-cars!”
  14. 14. Use-cases (contd.) … <ul><li>Search by category </li></ul><ul><ul><li>User wants to retrieve all visual content in a particular category </li></ul></ul><ul><ul><li>Difficult problem! </li></ul></ul><ul><ul><li>Semantic gap : gap between the user’s understanding and the features computed by a machine </li></ul></ul>Search by category …
  15. 15. Applications of visual search <ul><li>Art galleries & museum management </li></ul><ul><li>Searching product catalogs </li></ul><ul><li>Architectural & engineering design </li></ul><ul><li>Geographical information systems </li></ul><ul><li>Picture archiving </li></ul><ul><li>Law-enforcement & criminal investigations </li></ul>
  16. 16. Applications of Visual Search … for the art collector & museum curator
  17. 17. Product catalogs … … searching product catalogs
  18. 18. GIS systems … … searching GIS data/maps
  19. 19. Searching picture archives … personal image collections, online databases (Google, Creative Commons)
  20. 20. Law Enforcement … … searching faces/fingerprints/crime-scene photos
  21. 21. Applications in the mobile space … <ul><li>Camera phone + internet access </li></ul><ul><li>Endless possibilities!! </li></ul><ul><ul><li>Limited only by our imagination </li></ul></ul><ul><li>Here’s an example: </li></ul>
  22. 22. Visual search – Players in the mobile space An Overview:
  23. 23. Challenge – Sensory gap Limitation imposed by the sensor – not all information captured!
  24. 24. Challenge – Semantic Gap Difference between user-interpretation and machine-computed features
  25. 25. Visual search a.k.a. Content Based Image Retrieval (CBIR) Block-diagram of a typical content-based image retrieval system Comparison Query Index Index File Result Image database
  26. 26. Components of a CBIR system <ul><li>Image content descriptor </li></ul><ul><ul><li>Compute machine-understandable attributes </li></ul></ul><ul><li>Similarity/distance metrics </li></ul><ul><ul><li>Measure the similarity (or lack thereof) between query and database sample </li></ul></ul><ul><li>Indexing schemes </li></ul><ul><ul><li>How do you efficiently search the image/visual content database </li></ul></ul><ul><li>Relevance feedback </li></ul><ul><ul><li>Use the users’ choices to improve retrieval </li></ul></ul><ul><li>Performance measurement </li></ul><ul><ul><li>Metrics to measure effectiveness </li></ul></ul>
  27. 27. Image content descriptors <ul><li>Different attributes are used </li></ul><ul><ul><li>Color </li></ul></ul><ul><ul><li>Shape </li></ul></ul><ul><ul><li>Texture </li></ul></ul><ul><ul><li>Spatial layout </li></ul></ul>
  28. 28. Image content descriptors – Color
  29. 29. Image content descriptors – Color
  30. 30. Image content descriptors – Color <ul><li>Used extensively for image retrieval </li></ul><ul><ul><li>Motivation: human visual system </li></ul></ul><ul><ul><li>Simple & intuitive! </li></ul></ul><ul><ul><li>Not very discriminative </li></ul></ul><ul><ul><li>Used as a first pass to filter out unlikely examples </li></ul></ul>
  31. 31. Image descriptors – Color <ul><li>Color descriptors </li></ul><ul><ul><li>Color histograms – local/global </li></ul></ul><ul><ul><li>Color moments </li></ul></ul><ul><ul><li>Color coherence vector </li></ul></ul><ul><ul><li>Color correlogram </li></ul></ul>
  32. 32. Problem with color descriptors …
  33. 33. Image descriptors – Texture <ul><li>Different scenes may have same color! </li></ul><ul><li>Taking a cue from the human visual system (HVS) </li></ul>
  34. 34. Image descriptors – Texture Texture differentiates between a Lawn and a Forest
  35. 35. Image descriptors – Texture <ul><li>Wavelet transform features </li></ul><ul><ul><li>Multi-resolution approach to texture analysis </li></ul></ul><ul><ul><li>Texture described at various scales </li></ul></ul><ul><li>Gabor filter features </li></ul><ul><ul><li>Orientation and scale-tunable line (bar) detector </li></ul></ul><ul><li>Tamura features & Wold features </li></ul><ul><ul><li>Based on characteristics like coarseness, contrast, directionality, regularity (or lack thereof) </li></ul></ul>
  36. 36. Image descriptors - shape <ul><li>Segment foreground ‘objects’ </li></ul><ul><li>Shape can be used to describe these objects </li></ul><ul><li>Desirable attributes </li></ul><ul><ul><li>Should be invariant to translation, rotation and scaling </li></ul></ul>
  37. 37. Image descriptors – shape <ul><li>Classical shape representation uses moment variants </li></ul><ul><li>Boundary based methods </li></ul><ul><ul><li>Turning function or Turning angle </li></ul></ul><ul><li>Geometrical attributes </li></ul><ul><ul><li>Aspect ratios, (relative) dimensions </li></ul></ul>
  38. 38. Image descriptors – spatial information <ul><li>Sky is blue … but so is water! </li></ul><ul><li>What differentiates them is spatial layout! </li></ul><ul><li>Some common descriptors </li></ul><ul><ul><li>2D strings </li></ul></ul><ul><ul><li>Spatial quad-tree </li></ul></ul><ul><ul><li>Symbolic image </li></ul></ul>
  39. 39. The whole is greater than the sum of the parts!! <ul><li>Any one simple descriptor cannot give results required in ‘usable’ systems! </li></ul><ul><li>State-of-the-art systems use combination of descriptors </li></ul>
  40. 40. Similarity/distance measures <ul><li>Exact match cannot be found! </li></ul><ul><li>Similarity/distance measured by </li></ul><ul><ul><li>Quadratic-form distance </li></ul></ul><ul><ul><li>Mahalanobis distance </li></ul></ul><ul><ul><li>Minkowski-form distance </li></ul></ul><ul><ul><li>Histogram intersection </li></ul></ul><ul><ul><li>Kullback-Leibler divergence </li></ul></ul>
  41. 41. Similarity/distance metrics No one-size-fits-all!! Target application decides the metric to be used!
  42. 42. Indexing scheme <ul><li>Features typically have high dimensionality </li></ul><ul><li>Efficient indexing becomes a critical performance issue </li></ul><ul><ul><li>Dimensionality reduction (e.g., PCA) </li></ul></ul><ul><ul><li>R-tree, linear quad-trees, K-d-B-Tree, grid files </li></ul></ul>
  43. 43. Performance Evaluation <ul><li>Precision </li></ul><ul><ul><li>Precision is the fraction of retrieved images that are indeed relevant to the query </li></ul></ul><ul><li>Recall </li></ul><ul><ul><li>Recall is the fraction of relevant images returned by the query </li></ul></ul><ul><li>Trade-off between precision and recall </li></ul><ul><ul><li>Recall tends to increase as # retrieved items increases; precision decreases </li></ul></ul>
  44. 44. Performance Evaluation
  45. 45. Gazopa – A Visual Search Engine A ‘colorful’ example ;)
  46. 46. Gazopa – apples are red … Or are they?
  47. 47. Man’s best friend …
  48. 48. The sky is the limit …
  49. 49. iLike… searching products
  50. 50. The right descriptor is important! This guy’s not going to be happy! OMG!!!
  51. 51. Bing Image Search Am I looking for motorcycles, logos, Company?!
  52. 52. Thank You