Off the beaten track Using content-based multimedia similarity search for learning Suzanne Little Knowledge Media Institut...
Outline <ul><li>Content-Based Multimedia Similarity (CBMS)
What can CBMS be used for?
How does CBMS work?
CBMS for interactive learning </li></ul>
Content-Based Media Search Find visually similar images (BEWARE! Not always semantic similarity …) Goals: <ul><ul><li>Near...
Known object identification
General Search </li></ul></ul>?
Media -> Features Fingerprint <ul>[[169, 219, 175, 34, 76, 249, 55, 212, 12, ... 55, 78, 151, 78, 127, 224, 249, 47, 31, 5...
Similarity
Content-based Media Search <ul><li>Find a way to automatically generate a description of your media (preferably numerical!)
Calculate the “similarity” score between your query and the images you are searching
Return a ranked list of results </li></ul>
CBMS Applications <ul><li>Identify an object
Find instances of re-use
Discover and/or exposit (undirected search) </li></ul>
Spot&Search
Spot&Search Scott Forrest : E=MC squared &quot;Between finished surface texture and raw quarried stone. Between hard mater...
 
SocialLearn+Media <ul><li>4 months working with the SocialLearn project
Implement Content-Based Media Search as a web service </li></ul><ul><li>Find educational material by it's  visual similarity
What could I use? I don't understand? How are other people using the concept in different contexts? </li></ul>
SocialLearn+Media
SocialLearn+Media
Outline <ul><li>Content-Based Multimedia Similarity (CBMS)
What can CBMS be used for?
How does CBMS work?
Upcoming SlideShare
Loading in...5
×

Using content-based multimedia similarity search for learning

240

Published on

Talk given at the University of Middlesex, London to the Interaction Design Group, 2011-11-01.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
240
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Using content-based multimedia similarity search for learning

  1. 1. Off the beaten track Using content-based multimedia similarity search for learning Suzanne Little Knowledge Media Institute The Open University 1 st November 2011
  2. 2. Outline <ul><li>Content-Based Multimedia Similarity (CBMS)
  3. 3. What can CBMS be used for?
  4. 4. How does CBMS work?
  5. 5. CBMS for interactive learning </li></ul>
  6. 6. Content-Based Media Search Find visually similar images (BEWARE! Not always semantic similarity …) Goals: <ul><ul><li>Near-duplicate Detection
  7. 7. Known object identification
  8. 8. General Search </li></ul></ul>?
  9. 9. Media -> Features Fingerprint <ul>[[169, 219, 175, 34, 76, 249, 55, 212, 12, ... 55, 78, 151, 78, 127, 224, 249, 47, 31, 55, 45],[87, 222, 43, 83, 108, 9, 96, 201, 213, 71, ... 253, 12, 5, 239, 161, 209, 178, 80, 49, 177], [178, 235, 254, 136, 127, 168, 2, 126, 51, 31, 26, 131, 6, 70, 43, 19, 189, 179, 179, 208]] </ul>
  10. 10. Similarity
  11. 11. Content-based Media Search <ul><li>Find a way to automatically generate a description of your media (preferably numerical!)
  12. 12. Calculate the “similarity” score between your query and the images you are searching
  13. 13. Return a ranked list of results </li></ul>
  14. 14. CBMS Applications <ul><li>Identify an object
  15. 15. Find instances of re-use
  16. 16. Discover and/or exposit (undirected search) </li></ul>
  17. 17. Spot&Search
  18. 18. Spot&Search Scott Forrest : E=MC squared &quot;Between finished surface texture and raw quarried stone. Between hard materials and soft concepts. Between text and context.&quot; More information
  19. 20. SocialLearn+Media <ul><li>4 months working with the SocialLearn project
  20. 21. Implement Content-Based Media Search as a web service </li></ul><ul><li>Find educational material by it's visual similarity
  21. 22. What could I use? I don't understand? How are other people using the concept in different contexts? </li></ul>
  22. 23. SocialLearn+Media
  23. 24. SocialLearn+Media
  24. 25. Outline <ul><li>Content-Based Multimedia Similarity (CBMS)
  25. 26. What can CBMS be used for?
  26. 27. How does CBMS work?
  27. 28. CBMS for interactive learning </li></ul>
  28. 29. Searching with features <ul><li>Distance
  29. 30. Speed
  30. 31. Memory
  31. 32. Precision </li></ul>Q
  32. 33. SIFT <ul>E.g. <ul><li>992 keypoints
  33. 34. Each described by 128 numbers </li></ul></ul><ul><li>Scale Invariant Feature Transforms
  34. 35. Local features – invariant to changes in scale and transformation (e.g., colour, rotation) </li></ul>Lowe2004 - http://www.cs.ubc.ca/~lowe/keypoints/
  35. 36. Feature Space X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X SIFT has 128 dimensions X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
  36. 37. Results Rank images by frequency of occurrence in the neighbourhoods of a query's keypoints Each nn is a keypoint of an image in the index, I. Q {x 1 , x 2 ,…, x n } SIFT Nearest Neighbour Index (FLANN) x 1 ->{nn 1 , nn 2 , ..., nn K } x 2 ->{nn 1 , nn 2 , ..., nn K } x n ->{nn 1 , nn 2 , ..., nn K } … I
  37. 38. Ranked list of results 1 2 n … Object identification? <ul><ul><li>Use label from Rank 1 (transfer)
  38. 39. Merge results from same object and vote </li></ul></ul>Search? <ul><ul><li>Return top N results </li></ul></ul>Q
  39. 40. Evaluation <ul><li>Spot&Search – OU Art collection </li><ul><li>17 objects, 73 query images, 81 indexed images
  40. 41. Precision at 1: 0.85 (62 correct objects) </li></ul><li>UK bench collection (random subset) </li><ul><li>500 sets of 4 images (of same object)
  41. 42. Precision at 4: 0.77 </li></ul></ul>
  42. 43. Confidence <ul><li>Useful to filter results
  43. 44. Users like to know why </li></ul>
  44. 45. Confidence
  45. 46. Using media search <ul><li>What am I looking at?
  46. 47. I've got a screenshot – where does it come from? What is it's context?
  47. 48. I've got to prepare a course on Rome. I need some illustrations.
  48. 49. I don't understand this slide – find alternatives or discussions </li></ul>How can multimedia content-based search help?
  49. 50. Practical issues <ul><li>You can't find it if it's not indexed </li><ul><li>Garbage in = garbage out </li></ul><li>Visual similarity ↛ Semantic similarity
  50. 51. Users like to know why (they're more forgiving!)
  51. 52. Tradeoff – speed vs accuracy </li></ul>
  52. 53. What could be next? <ul><li>Technical: </li><ul><li>Post-search processing and filtering
  53. 54. More sophisticated nearest neighbour analysis
  54. 55. Efficiency improvements </li></ul><li>Expanding Spot&Search content (buildings?)
  55. 56. Indexing different media types </li><ul><li>How to intelligently handle diagrams?
  56. 57. What about text? </li></ul><li>Incorporating user feedback </li></ul>
  57. 58. Summary <ul><li>New ways </li><ul><li>to present information
  58. 59. to interact with material
  59. 60. to identify interesting stuff </li></ul></ul>How can multimedia content-based search help?
  60. 61. Acknowledgements and links <ul><li>Spot&Search - http://spotandsearch.kmi.open.ac.uk </li><ul><li>Adam Rae for photography
  61. 62. Paul Hogan for iPhone development
  62. 63. Jon Linney for graphic design </li></ul><li>SocialLearn - http://sociallearn.org </li><ul><li>Thanh Le and Darius Augaitis for integration with the SocialLearn toolbar </li></ul></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×