Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

User Interface for an Image Retrieval Engine System

481 views

Published on

https://imatge.upc.edu/web/publications/demonstration-open-source-framework-qualitative-evaluation-cbir-systems

Evaluating image retrieval systems in a quantitative way, for example by computing measures like mean average precision, allows for objective comparisons with a ground-truth. However, in cases where ground-truth is not available, the only alternative is to collect feedback from a user. Thus, qualitative assessments become important to better understand how the system works. Visualizing the results could be, in some scenarios, the only way to evaluate the results obtained and also the only opportunity to identify that a system is failing. This necessitates developing a User Interface (UI) for a Content Based Image Retrieval (CBIR) system that allows visualization of results and improvement via capturing user relevance feedback. A well-designed UI facilitates understanding of the performance of the system, both in cases where it works well and perhaps more importantly those which highlight the need for improvement. Our open-source system implements three components to facilitate researchers to quickly develop these capabilities for their retrieval engine. We present: a web-based user interface to visualize retrieval results and collect user annotations; a server that simplifies connection with any underlying CBIR system; and a server that manages the search engine data.

Published in: Data & Analytics
  • Be the first to comment

User Interface for an Image Retrieval Engine System

  1. 1. UI for an image retrieval engine system Paula Gomez Duran A project carried out in The Insight Centre for Data Analytics , in DCU Kevin McGuinness, Eva Mohedano, Xavier Giró-i-Nieto
  2. 2. Content Based Image Retrieval (CBIR) 2 Datasets Visual Query Expected outcome: “A dog”
  3. 3. Why is a UI useful for CBIR ? ● The importance of visualizing the results ● Ability to capture the user's intent 33
  4. 4. Contributions to the project - Development of the UI - Incorporate to the system different modes of interaction - Quantitative and qualitative evaluation 4
  5. 5. Contributions to the project - Development of the UI - Incorporate to the system different modes of interaction - Quantitative and qualitative evaluation 5
  6. 6. Interconnecting the system DEVELOPING UI 6… choosing tools to develop the project was the next step...
  7. 7. ReactJs | NodeJS | Python ● Framework of JavaScript ● Scalability, speed and simplicity. ● Fast ● Virtual DOM ● Fast and scalable network apps ● Single-thread using non-blocking I/O calls ● Capable of handling huge number of simultaneous connections with high throughput. ● NOT able to handle CPU-intensive operations ● Focuses on code readability ● Large standard libraries ● SLOW with speed in request or response petitions processing DEVELOPING UI 7
  8. 8. Interconnecting the system DEVELOPING UI ZERO_RPC 8
  9. 9. Image datasets DEVELOPING UI ● OXFORD BUILDING (5,063) ● PARIS BUILDING (6,412) ● INSTRE (28,543) Flickr multiple sources 9
  10. 10. Computing a ranking 10
  11. 11. 11
  12. 12. CBIR system ● All images analysed and stored ● Compare query analysed with all the other queries (cosine) ** Mohedano, Eva, Kevin McGuinness, Noel E. O'Connor, Amaia Salvador, Ferran Marques, and Xavier Giro-i-Nieto. "Bags of local convolutional features for scalable instance search." ACM ICMR, 2016. DEVELOPING UI 12
  13. 13. INPUTS OF THE SYSTEM URL IMAGE FROM FILE SYSTEM EXAMPLES DEVELOPING UI 13
  14. 14. INPUTS OF THE SYSTEM DEVELOPING UI URL SYSTEM EXAMPLES IMAGE FROM FILE 14
  15. 15. Contributions to the project - Development of the UI - Incorporate to the system different modes of interaction - Quantitative and qualitative evaluation 15
  16. 16. Functionalities of the system ● Explorer mode INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION ● Query expansion mode ● Annotation mode 16
  17. 17. Explorer mode : - motivation → Get to know the datasets and explore the system - functioning → When the first query is received and the ranking of the similar images is computed, whichever other image appearing below can be selected now as the new query to search into the dataset. INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION 17
  18. 18. Query expansion mode : - motivation → Get to know how the algorithm works in the system. - functioning → Average of the multiple image descriptors selected providing richer representation. INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION 18
  19. 19. Annotation mode : - motivation → Improve the accuracy of the automatic system by user’s interaction. - functioning → Annotating images and submit to the system to it can train an SVM. INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION 19
  20. 20. Contributions to the project - Development of the UI - Incorporate to the system different modes of interaction - Quantitative and qualitative evaluation 20
  21. 21. Feedback of users ● UI intuitive ● UI robust and consistent ● UI fully featured ● Understand the purpose of the UI ● Understanding modes with existing explanations ● Explorer mode useful regarding the ‘clickable’ function ● Query expansion mode useful to experiment without affecting the systems accuracy . ● Annotation mode useful to improve accuracy of a trained model. Questionnaire data represented in a graphic Strongly agree Agree QUANTITATIVE AND QUALITATIVE EVALUATION 80% 100% 80% 80% 70% 70% 90% 90% 21
  22. 22. Query expansion mode: QUANTITATIVE AND QUALITATIVE EVALUATION ● Low Average Precision → can improve the ranking ● High Average Precision → Just adds noise 22
  23. 23. Annotation mode : QUANTITATIVE AND QUALITATIVE EVALUATION ● Possibility to give just the negative feedback ● Possibility to train a model in order of improve the system by just annotating some images of the dataset. 23
  24. 24. CONCLUSIONS 24
  25. 25. Conclusions ● UI for an image retrieval system. ● User’s feedback was positive in the questionnaire done. ● UI works with 3 commonly used CBIR benchmarks : ❖ Oxford, Paris and Instre ● Annotation tool has been developed ● Quantitative and qualitative evaluation have been carried out. ● Structure in blocks → Can be adapted for other retrieval algorithms. 25
  26. 26. FUTURE WORK ● Include a 'Crop' mode on the query images to specify the region of interest. ● Unify structure of all datasets ● Include a mechanism to measure the time expand per query image. ● Include the ability to search within all photos in the three datasets. 26

×