Visual Search


Published on

Look at basics of image retrieval, some existing solutions and technology and challenges.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • TODO: Update this slide
  • Searching for the right image can be like searching for a needle in a haystack! Picture speaks better than thousand words, but how do you find the right pictures in the thousands (hundreds of thousands!) of pictures?!
  • Growth of digital content is exponential; we need a way to index and search this content.
  • Doing it well, and doing it fast is the challenge!
  • Where does the search query come from?
  • Mobile phone + internet access + search databases = Awesomeness!
  • Include something on mobile space, this is esp. relevant to Samsung
  • Mobile visual search: SnapTell: enhanced print-advertising; index covers books, CDs/DVDs, games Kooabs: Spin-off from ETH, Zurich (Luc Van Gool is one of the founders!) Idee: TinEye Mobile app for the iPhone Evolution robotics: primary licensee of D. Lowe’s SIFT patent
  • Visual Search

    1. 1. Visual Search aka Content-based Image Retrieval Amit Prabhudesai, SISO
    2. 2. Outline <ul><li>What is Visual search? </li></ul><ul><ul><li>Why do we need it? </li></ul></ul><ul><li>Use-cases & applications </li></ul><ul><li>Some existing solutions/technology </li></ul><ul><li>Basics of an Image Retrieval system </li></ul><ul><ul><li>Descriptors </li></ul></ul><ul><ul><li>Similarity measures </li></ul></ul><ul><ul><li>Indexing schemes </li></ul></ul><ul><li>How do you measure performance? </li></ul><ul><ul><li>How do existing solutions/technology fare? </li></ul></ul>
    3. 3. Why do we need visual search? How do I find what I’m looking for?!
    4. 4. Some telling numbers … <ul><li>Number of photos uploaded as of May2009 – 15 billion </li></ul><ul><li>220 million new photos per week </li></ul><ul><li>550,000 images served per second </li></ul><ul><li>is now the biggest photo-sharing site in the world! </li></ul>Source:
    5. 5. Searching Images is like …
    6. 6. What is visual search? Text query, textual results Text query, visual results Visual query, visual results … a.k.a. Visual Search
    7. 7. A Picture speaks better than … Visual search – why type when you can see it :
    8. 8. Visual search – specifying queries
    9. 9. Visual search – specifying queries
    10. 10. Visual search – specifying queries Snap a picture of something you like & upload it!
    11. 11. Visual search – use-cases <ul><li>Search by browsing </li></ul><ul><ul><li>User begins by submitting a keyword for the object-of-interest/uploading an image </li></ul></ul><ul><ul><li>System returns visual results (images/videos) </li></ul></ul><ul><ul><li>User browses through them and marks interesting results and asks system to return similar content </li></ul></ul>Search by browsing … a.k.a. “show me similar content”
    12. 12. Visual search – use-cases <ul><li>Search by drawing </li></ul><ul><ul><li>“ A picture speaks better than a thousand words”!! </li></ul></ul><ul><ul><li>User draws out an object/concept that she has in mind </li></ul></ul><ul><ul><li>System returns visual content similar to the object drawn </li></ul></ul>Search by drawing … Searching for LOVE ? ;)
    13. 13. Uses-cases … <ul><li>Search by example </li></ul><ul><ul><li>User has a specific query – e.g. she may be looking for red-cars </li></ul></ul><ul><ul><li>Uploads an example of the object-of-interest </li></ul></ul><ul><ul><li>System returns similar visual content </li></ul></ul>Search by example … a.k.a. “Show me all red-cars!”
    14. 14. Use-cases (contd.) … <ul><li>Search by category </li></ul><ul><ul><li>User wants to retrieve all visual content in a particular category </li></ul></ul><ul><ul><li>Difficult problem! </li></ul></ul><ul><ul><li>Semantic gap : gap between the user’s understanding and the features computed by a machine </li></ul></ul>Search by category …
    15. 15. Applications of visual search <ul><li>Art galleries & museum management </li></ul><ul><li>Searching product catalogs </li></ul><ul><li>Architectural & engineering design </li></ul><ul><li>Geographical information systems </li></ul><ul><li>Picture archiving </li></ul><ul><li>Law-enforcement & criminal investigations </li></ul>
    16. 16. Applications of Visual Search … for the art collector & museum curator
    17. 17. Product catalogs … … searching product catalogs
    18. 18. GIS systems … … searching GIS data/maps
    19. 19. Searching picture archives … personal image collections, online databases (Google, Creative Commons)
    20. 20. Law Enforcement … … searching faces/fingerprints/crime-scene photos
    21. 21. Applications in the mobile space … <ul><li>Camera phone + internet access </li></ul><ul><li>Endless possibilities!! </li></ul><ul><ul><li>Limited only by our imagination </li></ul></ul><ul><li>Here’s an example: </li></ul>
    22. 22. Visual search – Players in the mobile space An Overview:
    23. 23. Challenge – Sensory gap Limitation imposed by the sensor – not all information captured!
    24. 24. Challenge – Semantic Gap Difference between user-interpretation and machine-computed features
    25. 25. Visual search a.k.a. Content Based Image Retrieval (CBIR) Block-diagram of a typical content-based image retrieval system Comparison Query Index Index File Result Image database
    26. 26. Components of a CBIR system <ul><li>Image content descriptor </li></ul><ul><ul><li>Compute machine-understandable attributes </li></ul></ul><ul><li>Similarity/distance metrics </li></ul><ul><ul><li>Measure the similarity (or lack thereof) between query and database sample </li></ul></ul><ul><li>Indexing schemes </li></ul><ul><ul><li>How do you efficiently search the image/visual content database </li></ul></ul><ul><li>Relevance feedback </li></ul><ul><ul><li>Use the users’ choices to improve retrieval </li></ul></ul><ul><li>Performance measurement </li></ul><ul><ul><li>Metrics to measure effectiveness </li></ul></ul>
    27. 27. Image content descriptors <ul><li>Different attributes are used </li></ul><ul><ul><li>Color </li></ul></ul><ul><ul><li>Shape </li></ul></ul><ul><ul><li>Texture </li></ul></ul><ul><ul><li>Spatial layout </li></ul></ul>
    28. 28. Image content descriptors – Color
    29. 29. Image content descriptors – Color
    30. 30. Image content descriptors – Color <ul><li>Used extensively for image retrieval </li></ul><ul><ul><li>Motivation: human visual system </li></ul></ul><ul><ul><li>Simple & intuitive! </li></ul></ul><ul><ul><li>Not very discriminative </li></ul></ul><ul><ul><li>Used as a first pass to filter out unlikely examples </li></ul></ul>
    31. 31. Image descriptors – Color <ul><li>Color descriptors </li></ul><ul><ul><li>Color histograms – local/global </li></ul></ul><ul><ul><li>Color moments </li></ul></ul><ul><ul><li>Color coherence vector </li></ul></ul><ul><ul><li>Color correlogram </li></ul></ul>
    32. 32. Problem with color descriptors …
    33. 33. Image descriptors – Texture <ul><li>Different scenes may have same color! </li></ul><ul><li>Taking a cue from the human visual system (HVS) </li></ul>
    34. 34. Image descriptors – Texture Texture differentiates between a Lawn and a Forest
    35. 35. Image descriptors – Texture <ul><li>Wavelet transform features </li></ul><ul><ul><li>Multi-resolution approach to texture analysis </li></ul></ul><ul><ul><li>Texture described at various scales </li></ul></ul><ul><li>Gabor filter features </li></ul><ul><ul><li>Orientation and scale-tunable line (bar) detector </li></ul></ul><ul><li>Tamura features & Wold features </li></ul><ul><ul><li>Based on characteristics like coarseness, contrast, directionality, regularity (or lack thereof) </li></ul></ul>
    36. 36. Image descriptors - shape <ul><li>Segment foreground ‘objects’ </li></ul><ul><li>Shape can be used to describe these objects </li></ul><ul><li>Desirable attributes </li></ul><ul><ul><li>Should be invariant to translation, rotation and scaling </li></ul></ul>
    37. 37. Image descriptors – shape <ul><li>Classical shape representation uses moment variants </li></ul><ul><li>Boundary based methods </li></ul><ul><ul><li>Turning function or Turning angle </li></ul></ul><ul><li>Geometrical attributes </li></ul><ul><ul><li>Aspect ratios, (relative) dimensions </li></ul></ul>
    38. 38. Image descriptors – spatial information <ul><li>Sky is blue … but so is water! </li></ul><ul><li>What differentiates them is spatial layout! </li></ul><ul><li>Some common descriptors </li></ul><ul><ul><li>2D strings </li></ul></ul><ul><ul><li>Spatial quad-tree </li></ul></ul><ul><ul><li>Symbolic image </li></ul></ul>
    39. 39. The whole is greater than the sum of the parts!! <ul><li>Any one simple descriptor cannot give results required in ‘usable’ systems! </li></ul><ul><li>State-of-the-art systems use combination of descriptors </li></ul>
    40. 40. Similarity/distance measures <ul><li>Exact match cannot be found! </li></ul><ul><li>Similarity/distance measured by </li></ul><ul><ul><li>Quadratic-form distance </li></ul></ul><ul><ul><li>Mahalanobis distance </li></ul></ul><ul><ul><li>Minkowski-form distance </li></ul></ul><ul><ul><li>Histogram intersection </li></ul></ul><ul><ul><li>Kullback-Leibler divergence </li></ul></ul>
    41. 41. Similarity/distance metrics No one-size-fits-all!! Target application decides the metric to be used!
    42. 42. Indexing scheme <ul><li>Features typically have high dimensionality </li></ul><ul><li>Efficient indexing becomes a critical performance issue </li></ul><ul><ul><li>Dimensionality reduction (e.g., PCA) </li></ul></ul><ul><ul><li>R-tree, linear quad-trees, K-d-B-Tree, grid files </li></ul></ul>
    43. 43. Performance Evaluation <ul><li>Precision </li></ul><ul><ul><li>Precision is the fraction of retrieved images that are indeed relevant to the query </li></ul></ul><ul><li>Recall </li></ul><ul><ul><li>Recall is the fraction of relevant images returned by the query </li></ul></ul><ul><li>Trade-off between precision and recall </li></ul><ul><ul><li>Recall tends to increase as # retrieved items increases; precision decreases </li></ul></ul>
    44. 44. Performance Evaluation
    45. 45. Gazopa – A Visual Search Engine A ‘colorful’ example ;)
    46. 46. Gazopa – apples are red … Or are they?
    47. 47. Man’s best friend …
    48. 48. The sky is the limit …
    49. 49. iLike… searching products
    50. 50. The right descriptor is important! This guy’s not going to be happy! OMG!!!
    51. 51. Bing Image Search Am I looking for motorcycles, logos, Company?!
    52. 52. Thank You