Sketching Out Your Search Intent

1,321 views

Published on

A talk I gave at TPC workshop of ACM Multimedia 2010 at University of Amsterdam on June 24, 2010.

Published in: Technology, Travel
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,321
On SlideShare
0
From Embeds
0
Number of Embeds
35
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Sketching Out Your Search Intent

  1. 1. Sketching out Your Search Intent - Towards bridging intent and semantic gaps in web-scale multimedia search Xian-Sheng Hua Lead Researcher, Microsoft Research Asia June 24 2010 – ACM Multimedia 2010 TPC Meeting – University of Amsterdam
  2. 2. Xian-Sheng Hua, MSR Asia Evolution of CBIR/CBVR Annotation Based Commercial Search ~2006  Engines ~2003 (1) Tagging Based (2) Large-Scale CBIR/CBVR Conventional CBIR ~2001 Direct-Text Based 1990’ Academia Prototypes 1970~ 1980’ Manual Labeling Based 2
  3. 3. Xian-Sheng Hua, MSR Asia Three Schemes Example-Based Annotation-Based Text-Based
  4. 4. Xian-Sheng Hua, MSR Asia Limitation of Text-Based Search • High noises ̵ Surrounding text: frequently not related to the visual content ̵ Social tags: also noisy, and a large portion don’t have tags • Mostly only covers simple semantics ̵ Surrounding text: limited ̵ Social tags: People tend to only input simple and personal tags ̵ Query association: powerful but coverage is still limited (non-clicked images will never be well indexed) A complex example: Finding Lady Gaga walking on red carpet with black dress
  5. 5. Xian-Sheng Hua, MSR Asia Limitation of Annotation-Based Search • Low accuracy ̵ The accuracy of automatic annotation is still far from satisfactory ̵ Difficult to solve due • Limited coverage ̵ The coverage of the semantic concepts is still far from the rich content that is contain in an image ̵ Difficult to extend • High computation ̵ Learning costs high computation ̵ Recognition costs high if the number of concepts is large Still has a long way to go
  6. 6. Xian-Sheng Hua, MSR Asia Limitation of Large-Scale Example-Based Search • You need an example first ̵ How to do it if I don’t have an initial example? • Large-scale (semantic) similarity search is still difficult ̵ Though large-scale (near) duplicate detection is good An example: TinEye
  7. 7. Xian-Sheng Hua, MSR Asia Possible Way-Outs? • Large-scale high-performance annotation? ̵ Model-based? Data-driven? Hybrid? … • Large-scale manual labeling? ̵ ESP game? Label Me? Machinery Turk? … • New query interface? ̵ Interactive search? Or … To combine text, content and interface?
  8. 8. Xian-Sheng Hua, MSR Asia Simple Attempts • Exemplary attempts based on this idea ̵ Search by dominant color (Google/Bing) ̵ Show similar images (Bing)/Find similar images (Google) ̵ Show more sizes (Bing)
  9. 9. Xian-Sheng Hua, MSR Asia Two Recent Projects • Sketching out your search intent ̵ Image Search by Color Sketch ̵ Image Search by Concept Sketch
  10. 10. Search by Color Sketch WWW 2010 Demo
  11. 11. Xian-Sheng Hua, MSR Asia Seems It Is Not New? • “Query by sketch” has been proposed long time ago ̵ But on small datasets and difficult to scale up ̵ And seldom work well actually ̵ Not use to use – “object sketch”
  12. 12. It is based on a SIGGRAPH 95’s paper: C. Jacobs et al. Fast Multiresolution Image Querying. SIGGRAPH 1995 on small scale image set and difficult to scale-up (0.44 second for 10,000 images)
  13. 13. Xian-Sheng Hua, MSR Asia
  14. 14. Xian-Sheng Hua, MSR Asia Our Approach • Color sketch … ̵ Is not “object sketch” ̵ But only rough color distribution ̵ Supports large-scale data ̵ And uses text at the same time
  15. 15. Xian-Sheng Hua, MSR Asia Our Approach • Color sketch … ̵ Is not “object sketch” ̵ But only rough color distribution ̵ Supports large-scale data ̵ And uses text at the same time • Advantages of “Color Sketch” ̵ More presentative than dominant color (and text only) ̵ More robust than “object sketch” ̵ Easy to use compared with “object sketch” ̵ Easy to scale up
  16. 16. Xian-Sheng Hua, MSR Asia Example: Blue flower with green leaf
  17. 17. Xian-Sheng Hua, MSR Asia Example: Brooke Hogan walking on red carpet
  18. 18. Xian-Sheng Hua, MSR Asia Color Sketch Is More Powerful Blue Flower with Green Leaf Brooke Hogan walking on red carpet
  19. 19. Xian-Sheng Hua, MSR Asia Color Sketch Is More Powerful 30
  20. 20. Xian-Sheng Hua, MSR Asia Demo • It has been productized in Bing
  21. 21. Gambia flag
  22. 22. Xian-Sheng Hua, MSR Asia Summary – Why it Works • What are difficult ̵ Semantics is difficult ̵ Intent prediction is difficult • What are easier ̵ Use color to describe images’ content ̵ Use color to describe users’ intent content color sketch intent Color sketch is an intermediate representation to connect images’ content and users’ intent.
  23. 23. Xian-Sheng Hua, MSR Asia Limitation of Color Sketch • Only works well for those semantics that can be described by color distribution • Only works well when people is able to transfer their desire into color distribution A more complex example: A flying butterfly on the top-left of a flower.
  24. 24. Search by Concept Sketch SIGIR 2010
  25. 25. Xian-Sheng Hua, MSR Asia Search By Concept Sketch • A system for the image search intentions concerning: ̵ The presence of the semantic concepts (semantic constraints) ̵ the spatial layout of the concepts (spatial constraints) butterfly flower A concept map query Image search results of our system
  26. 26. Xian-Sheng Hua, MSR Asia Framework A concept map query A candidate image 1 A: butterfly Create Candidate B: flower Image Set butterfly A: X=0.5, Y=0.5 B: X=0.8, flower Y=0.8 2 3 Instant Concept Concept Distribution Modeling Estimation Concept Models Estimated distributions butterfly butterfly 4 flower Intent flower Interpretation Desired distributions 5 Spatial distribution butterfly comparison flower Relevance score
  27. 27. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “snoopy appear at right” snoopy Text-based image search “show similar image” snoopy Image search by concept map
  28. 28. Xian-Sheng Hua, MSR Asia Search Results Task: searching for the images with “snoopy appear at top” / “snoopy appear at left”
  29. 29. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “a jeep shown above a piece of grass” jeep grass
  30. 30. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “a car parked in front of a house” house car
  31. 31. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “a keyboard and a mouse being side by side” keyboard mouse
  32. 32. Xian-Sheng Hua, MSR Asia Search Results Comparison Task: searching for the images with “sky/Colosseum/grass appear from top to bottom” Colosseum grass
  33. 33. Xian-Sheng Hua, MSR Asia Advanced Function: Influence Scope Indication • Allow users to explicitly specify the occupied region of a concept house Default house lawn lawn house lawn Search for the long narrow lawn at the bottom of the image
  34. 34. Xian-Sheng Hua, MSR Asia Advanced Function: Influence Scope Adjustment Searching for small windmill Searching for large windmill
  35. 35. Xian-Sheng Hua, MSR Asia Advanced Function: Visual Modeling Adjustment sky house lawn Clear Search Advanced Function Visual Assistor Previous Next Allow users to indicate or narrow down what a desired concept looks like (visual constraints)
  36. 36. Xian-Sheng Hua, MSR Asia Advanced Function: Visual Modeling Adjustment Searching for front-view jeeps jeep Searching for side-view jeeps Visual instances
  37. 37. Xian-Sheng Hua, MSR Asia Advanced Function: Visual Modeling Adjustment Searching for butterfly with yellow flower butterfly flower Searching for butterfly with red flower Visual instances
  38. 38. Xian-Sheng Hua, MSR Asia Conclusions • To bridge the gap between users’ intent and visual semantics of images by ̵ Sketching out your search intent, and ̵ Combining text and visual content • Two exemplary approaches introduced ̵ Search by color sketch ̵ Search by concept sketch
  39. 39. Xian-Sheng Hua, MSR Asia Thank You

×