https://imatge.upc.edu/web/publications/livre-video-extension-lire-content-based-image-retrieval-system This project explores the expansion of Lucene Image Retrieval Engine (LIRE), an open-source Content-Based Image Retrieval (CBIR) system, for video retrieval on large scale video datasets. The fast growth of the need to store huge amounts of video in servers requires efficient, scalable search and indexing engines capable to assist users in their management and retrieval. In our tool, queries are formulated by visual examples allowing users to find the videos and the moment of time when the query image is matched with. The video dataset used on this scenario comprise over 1,000 hours of different news broadcast channels. This thesis presents an extension and adaptation of Lire and its plugin for Solr, an open-source enterprise search platform from the Apache Lucene project, for video retrieval based on visual features, as well as a web-interface for users from different devices.

1. LIVRE: A VIDEO EXTENSION TO THE
LIRE CONTENT-BASED IMAGE
RETRIEVAL SYSTEM
Degree’s Final Project Dissertation
Telecommunications Engineering
Gabriel de Oliveira
Supervisors:
Assoc. Prof. Mathias Lux
Assoc. Prof. Xavier Giró

2. Outline of the Thesis
1. Introduction
i. Motivation
ii. Overview and previous work
2. Proposed solution: The LIvRE system
i. Parsing
ii. Indexing
iii.Retrieval
3. Validation
i. Dataset
- Stanford I2V Newscasts dataset
ii. Experiments
- Quantitative evaluation
- Qualitative evaluation - The thinking-aloud test.
4. Conclusions and Further Work
March – October 2015
Slide 2

3. Motivation
Goal: To develop an all-in-one open source system for CBVR.
• Server side requeriments:
• Fast
• Scalable
• Flexible
• Automated
• User interface requeriments:
• Fast
• OS and device independent
• Mobile
Slide 3
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

4. Overview and previous work
Slide 6
• Open source CBIR Library in Java
• Apache Lucene core
• Solr plugin
• Supports parsing, indexing and retrieval
• Global and local descriptors
• Web-based interface
[1] Mathias Lux. LIRE: Open source image retrieval in java. In Proceedings of the 21st ACM international conference on Multimedia, pages 843{846.ACM, 2013.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

5. Developed solution: The LIvRE CBVR system
Slide 7
CBVR system - concept and requirements.
database
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

6. Developed solution: The LIvRE CBVR system
Slide 7
User sideServer side
LIvRE CBVR system architecture.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

7. Block 1: Parsing
System Architecture - Parsing
Slide 8
1. Find videos in any given folder structure.
2. Extract keyframes from those videos.
3. Parse extracted keyframes with selected image descriptors.
- Color Layout, Edge Histogram, JCD and PHOG
4. Generate XML Documents with the Feature Vectors.
Tools are provided to:
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

8. Block 2: Indexing
Fig. System Architecture
Slide 8
1. Find XML Documents containing the Feature Vectors
(generated from Parsing Block).
2. Upload XML documents to Solr.
3. Commit changes in Solr core.
Tools are provided to:
Fig. System Architecture - Indexing
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

9. Block 3: Retrieval
System Architecture - Retrieval
Slide 8
1. Image search field.
2. Settings.
User web-based interface input:
Web-based user interface input as
displayed on small screen devices.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

10. Block 3: Retrieval
System Architecture - Retrieval
Slide 8
1. Image search field.
2. Settings.
User web-based interface input:
Web-based user interface input as
displayed on small screen devices.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

11. Block 3: Retrieval
Slide 8
1. Candidate videos displayed using HTML5.
2. Thumbnails with other similar frames.
3. Time refinement.
4. Video information.
User web-based results presentation:
System Architecture - Retrieval Retrieval results presentation for
small screen devices
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

12. Block 3: Retrieval
Slide 8
1. Candidate videos displayed using HTML5.
2. Thumbnails with other similar frames.
3. Time refinement and ranking.
4. Video information.
User web-based results presentation:
Fig. System Architecture - Retrieval Fig. Retrieval results presentation
for small screen devices
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

13. LIvRE CBVR system demo
Introduction · Overview · LIvRE CBVR system · Dataset · Experiments · Conclusions
Slide 9

14. Validation
Stanford I2V Dataset
 Freely available data set.
 Large (~1TB Video)
• 23,443 video clips
• Average video duration: 2,65min.
• Keyframes @1fps: 3,808,760
• Video hours: 1,035h
 Ground-truth
• 78 queries
Some query images and video frames
from the Stanford I2V dataset.
Slide 14
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

15. Validation
Experiments
LIvRE CBVR system tested with 2
different evaluation methods:
Slide 16
1
2
Quantitative evaluation
Qualitative evaluation
(Thinking-aloud Test)
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

16. Quantitative study:
• Use ground-truth provided with the dataset for:
• Scene Retrieval evaluation (finding the right video).
• Time Refinement evaluation (finding the right moment of time at the right video).
Qualitative study:
• Web-based user interface.
• Thinking-aloud Test (offline).
• Participants are expert and non expert users.
• 4 Non-expert users.
• 2 Expert users.
Slide 17
1
2
Validation
Experiments
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

17. Quantitative study
Slide 18
LIvRE
Quantitative study evaluation process from provided ground-truth
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

18. Quantitative study
Slide 18
1st Stage: Scene Retrieval
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

19. Quantitative study
Slide 18
2nd Stage: Temporal Refinement
Temporal Refinement results for 100k candidates
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

20. Qualitative study
Thinking-aloud Test
• Volunteer participants perform specific tasks with the web-based
user interface.
• LIvRE CBVR system is running locally (offline) on the machine.
• Participants show their thoughts in loud-voice.
• Sessions are recorded and evaluated.
Slide 18
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

21. Qualitative study
Thinking-aloud Test
Slide 19
Sample input query frames Screenshots from Thinking-aloud test 1
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions
Timing results for 50K candidates (in miliseconds)

22. Conclusions and Future Work
- New CBVR System, LIvRE, was developed as an
extension of LIRE.
- LIvRE is now a branch of the LIRE Solr Project.
Slide 28
Future work:
• Local image descriptors.
• Integration of sound descriptors.
• Simplified set-up and deployment.
• Demo paper at ICMR 2016.
• Add Video annotation tool.
• Integration with computer vision / deep learning projects.
Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

23. Thank you for your attention
Do you have any question?
8 October 2015
LIvRE: A Video Extension to the LIRE
Content-Based Image Retrieval System.
Gabriel de Oliveira