More details: https://imatge.upc.edu/web/publications/rapid-serial-visual-presentation-relevance-feedback-image-retrieval-eeg-signals Author: Sergi Porta Advisors: Eva Mohedano & Noel O'Connor (DCU) / Amaia Salavador & Xavier Giró-i-Nieto (UPC) This thesis explores the potential of relevance feedback for image retrieval using EEG signals for human-computer interaction. This project aims at studying the optimal parameters of a rapid serial visual presentation (RSVP) of frames from a video database when the user is searching for an object instance. The simulations reported in this thesis assess the trade-off between using a small or a large amount of images in each RSVP round that captures the user feedback. While short RSVP rounds allow a quick learning of the user intention from the system, RSVP rounds must also be long enough to let users generate the P300 EEG signals which are triggered by relevant images. This work also addresses the problem of how to distribute potential relevant and non-relevant images in a RSVP round to maximize the probabilities of displaying each relevant frame separated at least 1 second from another relevant frame, as this configuration generates a cleaner P300 EEG signal. The presented simulations are based on a realistic set up for video retrieval with a subset of 1,000 frames from the TRECVID 2014 Instance Search task

2. ➔
➔
➔
➔
◆
◆
◆
➔
➔

3. • Exponential increase of generated content..

4. • ...still growing...

5. • ...so that the challenge is in retrieving them.

6. HUMAN
COMPUTER

9. E. Mohedano, A.Salvador, S. Porta, K. McGuinness, X.Giro, G. Healey N. O’Connor “Exploring EEG for Object
Detection and Retrieval” Submitted to International Conferece in Multimedia Retrieval 2015

10. List of N images to annotate

11. List of N annotated images per round
New list of N images to annotate
?

12. ●
●
●

13. ➔
➔
➔
➔
◆
◆
◆
➔
➔

14. Wang, J., Pohlmeyer, E., Hanna, B., Jiang, Y. G., Sajda, P., & Chang, S. F. (2009,
October). Brain state decoding for rapid image retrieval. In Proceedings of the 17th ACM
international conference on Multimedia (pp. 945-954). ACM.

15. E.Mohedano, G. Healy, K. McGuinness, X. Giro, N. O’Connor, A. Smeaton, Object Segmentation using
EEG Signals, 2014 ACMM, Florida (USA)

16. http://dl.acm.org/citation.cfm?id=500159
Simon Tong and Edward Chang. 2001. Support vector machine active learning for image retrieval.
In Proceedings of the ninth ACM international conference on Multimedia (MULTIMEDIA '01). ACM,
New York, NY, USA, 107-118. DOI=10.1145/500141.500159
● Releance Feedback

17. ➔
➔
➔
➔
◆
◆
◆
➔
➔

18. 18
Subset of TRECVID INS 2013 Dataset
● For each topic:
○ 4 images containing the object
○ 1000 images (database)
■ 50 relevant (targets) and 950 non-relevant (distractors)

19. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in
neural information processing systems (pp. 1097-1105).
Software:
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., ... & Darrell, T. (2014, November). Caffe: Convolutional
architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia(pp. 675-678). ACM. [web] 19

20. ➔
➔
➔
➔
◆
◆
◆
➔
➔

21. 21

22. 22
● Query Expansion
Average

23. 23
● Query Expansion

24. 24
● Query Expansion

25. 25
● Active Support Vector Machine

26. 26
Query Expansion ASVM
● Results
MAP

27. 27
● Which is the best search engine?

28. ➔
➔
➔
➔
◆
◆
◆
➔
➔

29. ...Rest... ...Rest Rest... Rest... ...
29
1 Round
● Which is the optimal round size?

30. 30
● Which is the optimal round size?

31. ➔
➔
➔
➔
◆
◆
◆
➔
➔

32. 32
● How to decide the images to show to the user?
● Highest Scores
● Random
● 5/95%
● Clusters

33. 33
Highest Scores 5/95%
● Distribution Strategy

34. 34
ClustersRandom
● Distribution Strategy

35. 35
● Distribution Strategy

36. 36
● EEG display

37. Visual evidence of discriminative P300 response on averaged epochs with
5x200 blocks @ 5Hz.
37

38. Unknown
...Rest...
Time between relevant (targets)
...Rest Rest... Rest... ...
38
● Less than 1 relevant/second:
○ RSVP @ 5Hz -- <20% relevants/round
○ RSVP @ 10Hz -- <10% relevants/round
1 Round

39. 39
● Distribution Strategy

40. 40
RSVP @ 5Hz
● EEG Display

41. 41
RSVP @ 10Hz
● EEG Display

42. ➔
➔
➔
➔
◆
◆
◆
➔
➔

43. ... ...
1 Round
25 frames
Active Support Vector Machine Optimal Size
K-means

44. ➔
➔
➔
➔
◆
◆
◆
➔
➔

45. 45

46. 46

47. 47
# €/hour Dedication
(week)
Salary N. Months Total
Junior
Engineer
1 8€ 20h 640€ 5 3200€
Senior
Engineer
2 20€ 5h 400€ 5 4000€
Phd
Researcher
2 8€ 5h 160€ 5 1600€
8800€

48. Evaluation Metric: Mean Average Precision