Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Towards Web-Scale
Content-Aware Image Search

             Xian-Sheng Hua
     Lead Researcher, Microsoft Research Asia

 ...
Xian-Sheng Hua, MSR Asia




About the Title
• Towards Web-Scale Content-Aware Image Search
  ̵   Towards …
  ̵   Web-Scal...
Xian-Sheng Hua, MSR Asia




Evolution of CBIR

                                      Annotation Based
  Commercial Search...
Xian-Sheng Hua, MSR Asia




Three Schemes

                Example-Based


            Annotation-Based


               ...
Xian-Sheng Hua, MSR Asia




Limitation of Text-Based Search

                    Surroundings




                       ...
Xian-Sheng Hua, MSR Asia




Limitation of Text-Based Search
• High noises
   ̵   Surrounding text: frequently not related...
Xian-Sheng Hua, MSR Asia




Limitation of Annotation-Based Search
• Low accuracy
  ̵   The accuracy of automatic annotati...
Xian-Sheng Hua, MSR Asia




Limitation of Large-Scale Example-Based Search
• You need an example first
   ̵   How to do i...
Xian-Sheng Hua, MSR Asia




Possible Way-Outs?
• Large-scale high-performance annotation?
   ̵   Model-based? Data-driven...
Xian-Sheng Hua, MSR Asia




Simple Attempts
• Exemplary attempts based on this idea
  ̵ Search by dominant color (Google/...
Xian-Sheng Hua, MSR Asia




Our Solutions: Two Recent Projects
• Image Search by Color Sketch
• Image Search by Concept S...
Search by Color Sketch
      WWW 2010 Demo
Xian-Sheng Hua, MSR Asia




Seems It Is Not New?
• “Query by sketch” has been proposed long time ago
  ̵   But on small d...
It is based on a SIGGRAPH 95’s paper:
       C. Jacobs et al. Fast Multiresolution Image Querying. SIGGRAPH 1995
on small ...
Xian-Sheng Hua, MSR Asia
Xian-Sheng Hua, MSR Asia




Our Approach
• Color sketch …
  ̵   Is not “object sketch”
  ̵   But only rough color distrib...
Xian-Sheng Hua, MSR Asia




Our Approach
• Color sketch …
  ̵   Is not “object sketch”
  ̵   But only rough color distrib...
Xian-Sheng Hua, MSR Asia




Example: Blue flower with green leaf
Xian-Sheng Hua, MSR Asia




Example: Brooke Hogan walking on red carpet
Xian-Sheng Hua, MSR Asia




Color Sketch Is More Powerful
  Blue Flower with Green Leaf   Brooke Hogan walking on red car...
Xian-Sheng Hua, MSR Asia




Color Sketch Is More Powerful




                                                  32
Xian-Sheng Hua, MSR Asia




  The Techniques
• Color Sketch: Image Representation
   ̵   Capturing both color and spatial...
Xian-Sheng Hua, MSR Asia




Color Sketch: Image Representation




Image with grids                         Color Sketch
...
Xian-Sheng Hua, MSR Asia




Intention Map:
From User’s Input to Intention Estimation




  Target: user’s input

        ...
Xian-Sheng Hua, MSR Asia




                           >
        Target   Image 1       Image 2




2010/6/24
Xian-Sheng Hua, MSR Asia




Intention Analysis 2: Relation



                           >
        Target   Image 1      ...
Xian-Sheng Hua, MSR Asia




Intention Analysis 3: Intention Propagation




                                             ...
Xian-Sheng Hua, MSR Asia




Intention Map: An Example




      User’s input

                                           ...
Xian-Sheng Hua, MSR Asia




Similarity Evaluation
• Image color map

• Intention map

• Similarity
Xian-Sheng Hua, MSR Asia




Experiment: Intention Justification
                                                    C
   ...
Xian-Sheng Hua, MSR Asia




Sensitivity to Grid Dimension
Xian-Sheng Hua, MSR Asia




Comparison
Xian-Sheng Hua, MSR Asia




Demo
• It has been productized in Bing
Gambia flag
Xian-Sheng Hua, MSR Asia




 Summary – Why it Works
• What are difficult
   ̵   Semantics is difficult
   ̵   Intent pred...
Xian-Sheng Hua, MSR Asia




Limitation of Color Sketch
• Only works well for those semantics that can
  be described by c...
Search by Concept Sketch
         SIGIR 2010
Xian-Sheng Hua, MSR Asia




Search By Concept Sketch
• A system for the image search intentions concerning:
   ̵   The pr...
Xian-Sheng Hua, MSR Asia




 Framework
A concept map query                                                           A ca...
Xian-Sheng Hua, MSR Asia




Creating Candidate Image Set
• Goal: get a group of candidate images from Web
     ̵   Ensure...
Xian-Sheng Hua, MSR Asia




Instant Concept Modeling
• A possible solution – manual tag-to-region




            Manual ...
Xian-Sheng Hua, MSR Asia




Instant Concept Modeling
• Goal: To learn a simple (but works well on the
  candidate image s...
Xian-Sheng Hua, MSR Asia




Concept Distribution Estimation
• Goal: Automatic concept localization leveraging Web-
  imag...
Xian-Sheng Hua, MSR Asia




Concept Distribution Estimation
  • Goal: estimate the distribution of a concept in an image
...
Xian-Sheng Hua, MSR Asia




Intention Interpretation
• Goal: represent user’s spatial intention as 2D distributions
• Pri...
Xian-Sheng Hua, MSR Asia




Spatial Distribution Comparison
• Goal: compute the similarity of the desired distributions a...
Xian-Sheng Hua, MSR Asia




Demo & Experiments
Xian-Sheng Hua, MSR Asia




 Search Results Comparison
  Task: searching for the images with “snoopy appear at right”



...
Xian-Sheng Hua, MSR Asia




Search Results
Task: searching for the images with “snoopy appear at top” / “snoopy appear at...
Xian-Sheng Hua, MSR Asia




Search Results Comparison
Task: searching for the images with “a jeep shown above a piece of ...
Xian-Sheng Hua, MSR Asia




Search Results Comparison
Task: searching for the images with “a car parked in front of a hou...
Xian-Sheng Hua, MSR Asia




Search Results Comparison
Task: searching for the images with “a keyboard and a mouse being s...
Xian-Sheng Hua, MSR Asia




Search Results Comparison
Task: searching for the images with “sky/Colosseum/grass appear fro...
Xian-Sheng Hua, MSR Asia




Quantitative Evaluation
Xian-Sheng Hua, MSR Asia




  Advanced Function:
  Influence Scope Adjustment
• Allow users to explicitly specify the occ...
Xian-Sheng Hua, MSR Asia




Advanced Function:
Influence Scope Adjustment



      Searching for small windmill




     ...
Xian-Sheng Hua, MSR Asia




Advanced Function:
Influence Scope Adjustment
Xian-Sheng Hua, MSR Asia




Advanced Function:
Visual Modeling Adjustment
                               sky


          ...
Xian-Sheng Hua, MSR Asia




Advanced Function:
Visual Modeling Adjustment


                Searching for
               ...
Xian-Sheng Hua, MSR Asia




Advanced Function:
Visual Modeling Adjustment


                  Searching for
             ...
Xian-Sheng Hua, MSR Asia




Conclusions
• The approaches of realizing web-scale content-
  aware image search
  ̵ Combing...
Xian-Sheng Hua, MSR Asia




Thank You
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Towards Web Scale Image Search
Upcoming SlideShare
Loading in …5
×

Towards Web Scale Image Search

1,090 views

Published on

A talk I was given in Delft University of Technology at Amsterdam, the Netherlands on June 23, 2010.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Towards Web Scale Image Search

  1. 1. Towards Web-Scale Content-Aware Image Search Xian-Sheng Hua Lead Researcher, Microsoft Research Asia June 23 2010 - TUDelft
  2. 2. Xian-Sheng Hua, MSR Asia About the Title • Towards Web-Scale Content-Aware Image Search ̵ Towards … ̵ Web-Scale … ̵ Content-Aware … ̵ Image Search …
  3. 3. Xian-Sheng Hua, MSR Asia Evolution of CBIR Annotation Based Commercial Search ~2006  Engines ~2003 (1) Tagging Based (2) Large-Scale CBIR/CBVR Conventional CBIR ~2001 Direct-Text Based 1990’ Academia Prototypes 1970~ 1980’ Manual Labeling Based 3
  4. 4. Xian-Sheng Hua, MSR Asia Three Schemes Example-Based Annotation-Based Text-Based
  5. 5. Xian-Sheng Hua, MSR Asia Limitation of Text-Based Search Surroundings Text Query Social Tags Association
  6. 6. Xian-Sheng Hua, MSR Asia Limitation of Text-Based Search • High noises ̵ Surrounding text: frequently not related to the visual content ̵ Social tags: also noisy, and a large portion don’t have tags • Mostly only covers simple semantics ̵ Surrounding text: limited ̵ Social tags: People tend to only input simple and personal tags ̵ Query association: powerful but coverage is still limited (non-clicked images will never be well indexed) A complex example: Finding Lady Gaga walking on red carpet and wearing in black
  7. 7. Xian-Sheng Hua, MSR Asia Limitation of Annotation-Based Search • Low accuracy ̵ The accuracy of automatic annotation is still far from satisfactory ̵ Difficult to be solved • Limited coverage ̵ The coverage of the semantic concepts is still far from the rich content that is contain in an image ̵ Difficult to extend • High computation ̵ Learning costs high computation ̵ Recognition costs high if the number of concepts is large Still has a long way to go
  8. 8. Xian-Sheng Hua, MSR Asia Limitation of Large-Scale Example-Based Search • You need an example first ̵ How to do it if I don’t have an initial example? • Large-scale (semantic) similarity search is still difficult ̵ Though large-scale (near) duplicate detection is good An example: TinEye
  9. 9. Xian-Sheng Hua, MSR Asia Possible Way-Outs? • Large-scale high-performance annotation? ̵ Model-based? Data-driven? Hybrid? … • Large-scale manual labeling? ̵ ESP game? Label Me? Machinery Turk? … • New query interface? ̵ Interactive search? Or … To combine text, content and interface?
  10. 10. Xian-Sheng Hua, MSR Asia Simple Attempts • Exemplary attempts based on this idea ̵ Search by dominant color (Google/Bing) ̵ Show similar images (Bing)/Find similar images (Google) ̵ Show more sizes (Bing)
  11. 11. Xian-Sheng Hua, MSR Asia Our Solutions: Two Recent Projects • Image Search by Color Sketch • Image Search by Concept Sketch
  12. 12. Search by Color Sketch WWW 2010 Demo
  13. 13. Xian-Sheng Hua, MSR Asia Seems It Is Not New? • “Query by sketch” has been proposed long time ago ̵ But on small datasets and difficult to scale up ̵ And seldom work well actually ̵ Not use to use – “object sketch”
  14. 14. It is based on a SIGGRAPH 95’s paper: C. Jacobs et al. Fast Multiresolution Image Querying. SIGGRAPH 1995 on small scale image set and difficult to scale-up (0.44 second for 10,000 images)
  15. 15. Xian-Sheng Hua, MSR Asia
  16. 16. Xian-Sheng Hua, MSR Asia Our Approach • Color sketch … ̵ Is not “object sketch” ̵ But only rough color distribution ̵ Supports large-scale data ̵ And uses text at the same time
  17. 17. Xian-Sheng Hua, MSR Asia Our Approach • Color sketch … ̵ Is not “object sketch” ̵ But only rough color distribution ̵ Supports large-scale data ̵ And uses text at the same time • Advantages of “Color Sketch” ̵ More presentative than dominant color (and text only) ̵ More robust than “object sketch” ̵ Easy to use compared with “object sketch” ̵ Easy to scale up
  18. 18. Xian-Sheng Hua, MSR Asia Example: Blue flower with green leaf
  19. 19. Xian-Sheng Hua, MSR Asia Example: Brooke Hogan walking on red carpet
  20. 20. Xian-Sheng Hua, MSR Asia Color Sketch Is More Powerful Blue Flower with Green Leaf Brooke Hogan walking on red carpet
  21. 21. Xian-Sheng Hua, MSR Asia Color Sketch Is More Powerful 32
  22. 22. Xian-Sheng Hua, MSR Asia The Techniques • Color Sketch: Image Representation ̵ Capturing both color and spatial information ̵ Low computation and storage costs • Intention Map: Intention Representation ̵ Rough: Tolerance to the errors in the colors and positions of the strokes ̵ Sparse: Intention estimation from sparse strokes • Similarity Evaluation ̵ Effective and fast
  23. 23. Xian-Sheng Hua, MSR Asia Color Sketch: Image Representation Image with grids Color Sketch Computation Cost: less than 50ms/image Storage Cost: 80 byte in average/image
  24. 24. Xian-Sheng Hua, MSR Asia Intention Map: From User’s Input to Intention Estimation Target: user’s input (1) Do not require users to input the color map exactly (2) Robust to rough and sparse input
  25. 25. Xian-Sheng Hua, MSR Asia > Target Image 1 Image 2 2010/6/24
  26. 26. Xian-Sheng Hua, MSR Asia Intention Analysis 2: Relation > Target Image 1 Image 2 2010/6/24
  27. 27. Xian-Sheng Hua, MSR Asia Intention Analysis 3: Intention Propagation < Propagated Image 1 Image 2 Since the target color sketch is rough, the cells nearby the scribbled cells may have larger chance to have the similar color. Hence, image 2 should be more similar as the blue color appearing in the cells near the scribbled cells. 2010/6/24
  28. 28. Xian-Sheng Hua, MSR Asia Intention Map: An Example User’s input Intention map for blue In scribbled cells, the intention weight is the largest, nearby cells has relatively smaller intention weight, but the cells scribbled in green and its nearby cells have negative intention weight for blue color.
  29. 29. Xian-Sheng Hua, MSR Asia Similarity Evaluation • Image color map • Intention map • Similarity
  30. 30. Xian-Sheng Hua, MSR Asia Experiment: Intention Justification C C+R C+P C+R+P C: Consistency R: Relation P: Propagation 51 text query, each 3 color sketches
  31. 31. Xian-Sheng Hua, MSR Asia Sensitivity to Grid Dimension
  32. 32. Xian-Sheng Hua, MSR Asia Comparison
  33. 33. Xian-Sheng Hua, MSR Asia Demo • It has been productized in Bing
  34. 34. Gambia flag
  35. 35. Xian-Sheng Hua, MSR Asia Summary – Why it Works • What are difficult ̵ Semantics is difficult ̵ Intent prediction is difficult • What are easier ̵ Use color to describe images’ content ̵ Use color to describe users’ intent content color sketch intent Color sketch is an intermediate representation to connect images’ content and users’ intent.
  36. 36. Xian-Sheng Hua, MSR Asia Limitation of Color Sketch • Only works well for those semantics that can be described by color distribution • Only works well when people are able to transfer their desire into color distribution A more complex example: A flying butterfly on the top-left of a flower.
  37. 37. Search by Concept Sketch SIGIR 2010
  38. 38. Xian-Sheng Hua, MSR Asia Search By Concept Sketch • A system for the image search intentions concerning: ̵ The presence of the semantic concepts (semantic constraints) ̵ the spatial layout of the concepts (spatial constraints) butterfly flower A concept map query Image search results of our system
  39. 39. Xian-Sheng Hua, MSR Asia Framework A concept map query A candidate image 1 A: butterfly Create Candidate B: flower Image Set butterfly A: X=0.5, Y=0.5 B: X=0.8, flower Y=0.8 2 3 Instant Concept Concept Distribution Modeling Estimation Concept Models Estimated distributions butterfly butterfly 4 flower Intent flower Interpretation Desired distributions 5 Spatial distribution butterfly comparison flower Relevance score
  40. 40. Xian-Sheng Hua, MSR Asia Creating Candidate Image Set • Goal: get a group of candidate images from Web ̵ Ensure the presences of the concepts ̵ Reduce computational cost butterfly flower flower butterfly A: butterfly Candidate Images: B: flower Images related to butterfly butterfly & flower flower Training data for concept “butterfly” Training data for concept “flower”
  41. 41. Xian-Sheng Hua, MSR Asia Instant Concept Modeling • A possible solution – manual tag-to-region Manual tagging is expensive
  42. 42. Xian-Sheng Hua, MSR Asia Instant Concept Modeling • Goal: To learn a simple (but works well on the candidate image set) visual detector for each concept butterfly [Visual instances] … Training data for Instance-based “butterfly” visual detector of “butterfly” Graph-based representative image selection
  43. 43. Xian-Sheng Hua, MSR Asia Concept Distribution Estimation • Goal: Automatic concept localization leveraging Web- image resources Web images A: A candidate image Instant Concept B: Distribution A: butterfly B: flower Modeling estimation Estimated distributions Advantages: Scalable to Web-scale concepts butterfly by exploiting rich Web-image resources flower
  44. 44. Xian-Sheng Hua, MSR Asia Concept Distribution Estimation • Goal: estimate the distribution of a concept in an image [Visual instances] … A visual instance A candidate image … … Estimated distributions of representative butterflies in the candidate image Estimated distribution
  45. 45. Xian-Sheng Hua, MSR Asia Intention Interpretation • Goal: represent user’s spatial intention as 2D distributions • Principles ̵ A concept should appear near the specified position ̵ A concept should not appear at the positions of the other concepts Desired distributions Desired distributions butterfly butterfly butterfly butterfly flower flower
  46. 46. Xian-Sheng Hua, MSR Asia Spatial Distribution Comparison • Goal: compute the similarity of the desired distributions and the estimated distributions Desired distributions Estimated distributions butterfly … butterfly flower … flower Distribution comparison Fusion Relevance score
  47. 47. Xian-Sheng Hua, MSR Asia Demo & Experiments
  48. 48. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “snoopy appear at right” snoopy Text-based image search “show similar image” snoopy Image search by concept map
  49. 49. Xian-Sheng Hua, MSR Asia Search Results Task: searching for the images with “snoopy appear at top” / “snoopy appear at left”
  50. 50. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “a jeep shown above a piece of grass” jeep grass
  51. 51. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “a car parked in front of a house” house car
  52. 52. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “a keyboard and a mouse being side by side” keyboard mouse
  53. 53. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “sky/Colosseum/grass appear from top to bottom” Colosseum grass
  54. 54. Xian-Sheng Hua, MSR Asia Quantitative Evaluation
  55. 55. Xian-Sheng Hua, MSR Asia Advanced Function: Influence Scope Adjustment • Allow users to explicitly specify the occupied region of a concept house Default house lawn lawn house lawn Search for the long narrow lawn at the bottom of the image
  56. 56. Xian-Sheng Hua, MSR Asia Advanced Function: Influence Scope Adjustment Searching for small windmill Searching for large windmill
  57. 57. Xian-Sheng Hua, MSR Asia Advanced Function: Influence Scope Adjustment
  58. 58. Xian-Sheng Hua, MSR Asia Advanced Function: Visual Modeling Adjustment sky house lawn Clear Search Advanced Function Visual Assistor Previous Next Allow users to indicate or narrow down what a desired concept looks like (visual constraints)
  59. 59. Xian-Sheng Hua, MSR Asia Advanced Function: Visual Modeling Adjustment Searching for front-view jeeps jeep Searching for side-view jeeps Visual instances
  60. 60. Xian-Sheng Hua, MSR Asia Advanced Function: Visual Modeling Adjustment Searching for butterfly with yellow flower butterfly flower Searching for butterfly with red flower Visual instances
  61. 61. Xian-Sheng Hua, MSR Asia Conclusions • The approaches of realizing web-scale content- aware image search ̵ Combing text and content ̵ Providing interface to express search intent ̵ Indexing and searching efficiently • Two exemplary approaches introduced ̵ Search by color sketch ̵ Search by concept sketch
  62. 62. Xian-Sheng Hua, MSR Asia Thank You

×