SlideShare a Scribd company logo
1 of 26
Download to read offline
UI for an image
retrieval engine
system
Paula Gomez Duran
A project carried out in The Insight Centre for Data Analytics , in DCU
Kevin McGuinness, Eva Mohedano, Xavier Giró-i-Nieto
Content Based Image Retrieval (CBIR)
2
Datasets
Visual Query
Expected outcome:
“A dog”
Why is a UI useful for CBIR ?
● The importance of
visualizing the results
● Ability to capture the
user's intent
33
Contributions to the project
- Development of the UI
- Incorporate to the system different modes of interaction
- Quantitative and qualitative evaluation
4
Contributions to the project
- Development of the UI
- Incorporate to the system different modes of interaction
- Quantitative and qualitative evaluation
5
Interconnecting the system
DEVELOPING UI
6… choosing tools to develop the project was the next step...
ReactJs | NodeJS | Python
● Framework of JavaScript
● Scalability, speed and simplicity.
● Fast
● Virtual DOM
● Fast and scalable network apps
● Single-thread using non-blocking I/O calls
● Capable of handling huge number of simultaneous
connections with high throughput.
● NOT able to handle CPU-intensive operations
● Focuses on code readability
● Large standard libraries
● SLOW with speed in request or
response petitions processing
DEVELOPING UI
7
Interconnecting the system
DEVELOPING UI
ZERO_RPC
8
Image datasets
DEVELOPING UI
● OXFORD BUILDING (5,063)
● PARIS BUILDING (6,412)
● INSTRE (28,543)
Flickr
multiple
sources
9
Computing a ranking
10
11
CBIR system
● All images analysed and stored
● Compare query analysed with all the other queries (cosine)
** Mohedano, Eva, Kevin McGuinness, Noel E. O'Connor, Amaia Salvador, Ferran Marques, and
Xavier Giro-i-Nieto. "Bags of local convolutional features for scalable instance search." ACM ICMR,
2016.
DEVELOPING UI
12
INPUTS OF THE SYSTEM
URL
IMAGE
FROM
FILE
SYSTEM
EXAMPLES
DEVELOPING UI
13
INPUTS OF THE SYSTEM
DEVELOPING UI
URL
SYSTEM
EXAMPLES
IMAGE
FROM
FILE
14
Contributions to the project
- Development of the UI
- Incorporate to the system different modes of interaction
- Quantitative and qualitative evaluation
15
Functionalities of the system
● Explorer mode
INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION
● Query expansion mode ● Annotation mode
16
Explorer mode :
- motivation → Get to know the datasets and explore the system
- functioning → When the first query is received and the ranking of the similar images is
computed, whichever other image appearing below can be selected now as the new query to
search into the dataset.
INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION
17
Query expansion mode :
- motivation → Get to know how the algorithm works in the system.
- functioning → Average of the multiple image descriptors selected providing richer representation.
INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION
18
Annotation mode :
- motivation → Improve the accuracy of the automatic system by user’s interaction.
- functioning → Annotating images and submit to the system to it can train an SVM.
INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION
19
Contributions to the project
- Development of the UI
- Incorporate to the system different modes of interaction
- Quantitative and qualitative evaluation
20
Feedback of users
● UI intuitive
● UI robust and consistent
● UI fully featured
● Understand the purpose of the UI
● Understanding modes with existing explanations
● Explorer mode useful regarding the ‘clickable’ function
● Query expansion mode useful to experiment without affecting
the systems accuracy .
● Annotation mode useful to improve accuracy of a trained model.
Questionnaire data represented in a graphic
Strongly agree
Agree
QUANTITATIVE AND QUALITATIVE EVALUATION
80%
100%
80%
80%
70%
70%
90%
90%
21
Query expansion mode:
QUANTITATIVE AND QUALITATIVE EVALUATION
● Low Average Precision → can improve the ranking
● High Average Precision → Just adds noise
22
Annotation mode :
QUANTITATIVE AND QUALITATIVE EVALUATION
● Possibility to give just the negative feedback
● Possibility to train a model in order of improve the
system by just annotating some images of the dataset.
23
CONCLUSIONS
24
Conclusions
● UI for an image retrieval system.
● User’s feedback was positive in the questionnaire done.
● UI works with 3 commonly used CBIR benchmarks :
❖ Oxford, Paris and Instre
● Annotation tool has been developed
● Quantitative and qualitative evaluation have been carried out.
● Structure in blocks → Can be adapted for other retrieval
algorithms.
25
FUTURE WORK
● Include a 'Crop' mode on the query images to specify the region of interest.
● Unify structure of all datasets
● Include a mechanism to measure the time expand per query image.
● Include the ability to search within all photos in the three datasets.
26

More Related Content

Similar to User Interface for an Image Retrieval Engine System

Software Engineering Process Models
Software Engineering Process Models Software Engineering Process Models
Software Engineering Process Models Satya P. Joshi
 
Ensemble Learning in Recommender Systems: Combining Multiple User Interaction...
Ensemble Learning in Recommender Systems: Combining Multiple User Interaction...Ensemble Learning in Recommender Systems: Combining Multiple User Interaction...
Ensemble Learning in Recommender Systems: Combining Multiple User Interaction...Arthur Fortes
 
Анализ атрибутов качества
Анализ атрибутов качестваАнализ атрибутов качества
Анализ атрибутов качестваSQALab
 
productionising-recommenders
productionising-recommendersproductionising-recommenders
productionising-recommendersLudovik Coba
 
Machine learning in production
Machine learning in productionMachine learning in production
Machine learning in productionTuri, Inc.
 
NIDHI PROJECT.pptx
NIDHI PROJECT.pptxNIDHI PROJECT.pptx
NIDHI PROJECT.pptxXavinr007
 
Preference Elicitation Interface
Preference Elicitation InterfacePreference Elicitation Interface
Preference Elicitation Interface晓愚 孟
 
Usability evaluation methods (part 2) and performance metrics
Usability evaluation methods (part 2) and performance metricsUsability evaluation methods (part 2) and performance metrics
Usability evaluation methods (part 2) and performance metricsAndres Baravalle
 
CYBERBULLYING DETECTION USING MACHINE LEARNING-1 (1).pdf
CYBERBULLYING DETECTION USING              MACHINE LEARNING-1 (1).pdfCYBERBULLYING DETECTION USING              MACHINE LEARNING-1 (1).pdf
CYBERBULLYING DETECTION USING MACHINE LEARNING-1 (1).pdfKumbidiGaming
 
Online movie ticket booking
Online movie ticket bookingOnline movie ticket booking
Online movie ticket bookingmrinnovater007
 
Multimodal interactions in recommender systems (Bracis 2014)
Multimodal interactions in recommender systems (Bracis 2014)Multimodal interactions in recommender systems (Bracis 2014)
Multimodal interactions in recommender systems (Bracis 2014)Arthur Fortes
 
HABIB FIGA GUYE {BULE HORA UNIVERSITY}(habibifiga@gmail.com
HABIB FIGA GUYE {BULE HORA UNIVERSITY}(habibifiga@gmail.comHABIB FIGA GUYE {BULE HORA UNIVERSITY}(habibifiga@gmail.com
HABIB FIGA GUYE {BULE HORA UNIVERSITY}(habibifiga@gmail.comHABIB FIGA GUYE
 
Performance Management of IT Service Processes Using a Mashup-based Approach
Performance Management of IT Service Processes Using a Mashup-based ApproachPerformance Management of IT Service Processes Using a Mashup-based Approach
Performance Management of IT Service Processes Using a Mashup-based ApproachCarlos Raniery
 
Performance evaluation of a multi-core system using Systems development meth...
 Performance evaluation of a multi-core system using Systems development meth... Performance evaluation of a multi-core system using Systems development meth...
Performance evaluation of a multi-core system using Systems development meth...Yoshifumi Sakamoto
 
Training Webinar: From a bad to an awesome user experience - Training Webinar
Training Webinar: From a bad to an awesome user experience - Training WebinarTraining Webinar: From a bad to an awesome user experience - Training Webinar
Training Webinar: From a bad to an awesome user experience - Training WebinarOutSystems
 

Similar to User Interface for an Image Retrieval Engine System (20)

Software Engineering Process Models
Software Engineering Process Models Software Engineering Process Models
Software Engineering Process Models
 
Ensemble Learning in Recommender Systems: Combining Multiple User Interaction...
Ensemble Learning in Recommender Systems: Combining Multiple User Interaction...Ensemble Learning in Recommender Systems: Combining Multiple User Interaction...
Ensemble Learning in Recommender Systems: Combining Multiple User Interaction...
 
Анализ атрибутов качества
Анализ атрибутов качестваАнализ атрибутов качества
Анализ атрибутов качества
 
productionising-recommenders
productionising-recommendersproductionising-recommenders
productionising-recommenders
 
sigir16
sigir16sigir16
sigir16
 
Machine learning in production
Machine learning in productionMachine learning in production
Machine learning in production
 
NIDHI PROJECT.pptx
NIDHI PROJECT.pptxNIDHI PROJECT.pptx
NIDHI PROJECT.pptx
 
Preference Elicitation Interface
Preference Elicitation InterfacePreference Elicitation Interface
Preference Elicitation Interface
 
Usability evaluation methods (part 2) and performance metrics
Usability evaluation methods (part 2) and performance metricsUsability evaluation methods (part 2) and performance metrics
Usability evaluation methods (part 2) and performance metrics
 
CYBERBULLYING DETECTION USING MACHINE LEARNING-1 (1).pdf
CYBERBULLYING DETECTION USING              MACHINE LEARNING-1 (1).pdfCYBERBULLYING DETECTION USING              MACHINE LEARNING-1 (1).pdf
CYBERBULLYING DETECTION USING MACHINE LEARNING-1 (1).pdf
 
Online movie ticket booking
Online movie ticket bookingOnline movie ticket booking
Online movie ticket booking
 
Sentiment Analysis
Sentiment AnalysisSentiment Analysis
Sentiment Analysis
 
Multimodal interactions in recommender systems (Bracis 2014)
Multimodal interactions in recommender systems (Bracis 2014)Multimodal interactions in recommender systems (Bracis 2014)
Multimodal interactions in recommender systems (Bracis 2014)
 
HABIB FIGA GUYE {BULE HORA UNIVERSITY}(habibifiga@gmail.com
HABIB FIGA GUYE {BULE HORA UNIVERSITY}(habibifiga@gmail.comHABIB FIGA GUYE {BULE HORA UNIVERSITY}(habibifiga@gmail.com
HABIB FIGA GUYE {BULE HORA UNIVERSITY}(habibifiga@gmail.com
 
Cavaros
CavarosCavaros
Cavaros
 
Performance Management of IT Service Processes Using a Mashup-based Approach
Performance Management of IT Service Processes Using a Mashup-based ApproachPerformance Management of IT Service Processes Using a Mashup-based Approach
Performance Management of IT Service Processes Using a Mashup-based Approach
 
Performance evaluation of a multi-core system using Systems development meth...
 Performance evaluation of a multi-core system using Systems development meth... Performance evaluation of a multi-core system using Systems development meth...
Performance evaluation of a multi-core system using Systems development meth...
 
Formal Definition of Collaborative Spaces
Formal Definition of Collaborative Spaces Formal Definition of Collaborative Spaces
Formal Definition of Collaborative Spaces
 
SE UNIT-2.pdf
SE UNIT-2.pdfSE UNIT-2.pdf
SE UNIT-2.pdf
 
Training Webinar: From a bad to an awesome user experience - Training Webinar
Training Webinar: From a bad to an awesome user experience - Training WebinarTraining Webinar: From a bad to an awesome user experience - Training Webinar
Training Webinar: From a bad to an awesome user experience - Training Webinar
 

More from Universitat Politècnica de Catalunya

The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...Universitat Politècnica de Catalunya
 
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoTowards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoUniversitat Politècnica de Catalunya
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Universitat Politècnica de Catalunya
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosGeneration of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosUniversitat Politècnica de Catalunya
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Universitat Politècnica de Catalunya
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Universitat Politècnica de Catalunya
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Universitat Politècnica de Catalunya
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Universitat Politècnica de Catalunya
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Universitat Politècnica de Catalunya
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Universitat Politècnica de Catalunya
 
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Universitat Politècnica de Catalunya
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Universitat Politècnica de Catalunya
 
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020Universitat Politècnica de Catalunya
 

More from Universitat Politècnica de Catalunya (20)

Deep Generative Learning for All
Deep Generative Learning for AllDeep Generative Learning for All
Deep Generative Learning for All
 
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
 
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoTowards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
 
The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
 
Open challenges in sign language translation and production
Open challenges in sign language translation and productionOpen challenges in sign language translation and production
Open challenges in sign language translation and production
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosGeneration of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
 
Discovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in MinecraftDiscovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in Minecraft
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...
 
Intepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural NetworksIntepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural Networks
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
 
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
 
Curriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object SegmentationCurriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object Segmentation
 
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
 
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
 

Recently uploaded

Empowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsEmpowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsGain Insights
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptxFurkanTasci3
 
Enabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsEnabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsNeo4j
 
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...ferisulianta.com
 
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdfNeo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdfNeo4j
 
Air Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfAir Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfJasonBoboKyaw
 
Microeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfMicroeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfmxlos0
 
Paul Martin (Gartner) - Show Me the AI Money.pdf
Paul Martin (Gartner) - Show Me the AI Money.pdfPaul Martin (Gartner) - Show Me the AI Money.pdf
Paul Martin (Gartner) - Show Me the AI Money.pdfdcphostmaster
 
Understanding the Impact of video length on student performance
Understanding the Impact of video length on student performanceUnderstanding the Impact of video length on student performance
Understanding the Impact of video length on student performancePrithaVashisht1
 
Báo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingBáo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingMarketingTrips
 
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j
 
Data Collection from Social Media Platforms
Data Collection from Social Media PlatformsData Collection from Social Media Platforms
Data Collection from Social Media PlatformsMahmoud Yasser
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTimothy Spann
 
PPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfggggPPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfggggbhadratanusenapati1
 
The market for cross-border mortgages in Europe
The market for cross-border mortgages in EuropeThe market for cross-border mortgages in Europe
The market for cross-border mortgages in Europe321k
 
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Neo4j
 
How to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product DevelopmentHow to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product DevelopmentAggregage
 
Data Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxData Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxEmmanuel Dauda
 
Unleashing Datas Potential - Mastering Precision with FCO-IM
Unleashing Datas Potential - Mastering Precision with FCO-IMUnleashing Datas Potential - Mastering Precision with FCO-IM
Unleashing Datas Potential - Mastering Precision with FCO-IMMarco Wobben
 

Recently uploaded (20)

Target_Company_Data_breach_2013_110million
Target_Company_Data_breach_2013_110millionTarget_Company_Data_breach_2013_110million
Target_Company_Data_breach_2013_110million
 
Empowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsEmpowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded Analytics
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
 
Enabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsEnabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge Graphs
 
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
Prediction Of Cryptocurrency Prices Using Lstm, Svm And Polynomial Regression...
 
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdfNeo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
 
Air Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfAir Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdf
 
Microeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfMicroeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdf
 
Paul Martin (Gartner) - Show Me the AI Money.pdf
Paul Martin (Gartner) - Show Me the AI Money.pdfPaul Martin (Gartner) - Show Me the AI Money.pdf
Paul Martin (Gartner) - Show Me the AI Money.pdf
 
Understanding the Impact of video length on student performance
Understanding the Impact of video length on student performanceUnderstanding the Impact of video length on student performance
Understanding the Impact of video length on student performance
 
Báo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingBáo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân Marketing
 
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
 
Data Collection from Social Media Platforms
Data Collection from Social Media PlatformsData Collection from Social Media Platforms
Data Collection from Social Media Platforms
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
 
PPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfggggPPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfgggg
 
The market for cross-border mortgages in Europe
The market for cross-border mortgages in EuropeThe market for cross-border mortgages in Europe
The market for cross-border mortgages in Europe
 
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
 
How to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product DevelopmentHow to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product Development
 
Data Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxData Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potx
 
Unleashing Datas Potential - Mastering Precision with FCO-IM
Unleashing Datas Potential - Mastering Precision with FCO-IMUnleashing Datas Potential - Mastering Precision with FCO-IM
Unleashing Datas Potential - Mastering Precision with FCO-IM
 

User Interface for an Image Retrieval Engine System

  • 1. UI for an image retrieval engine system Paula Gomez Duran A project carried out in The Insight Centre for Data Analytics , in DCU Kevin McGuinness, Eva Mohedano, Xavier Giró-i-Nieto
  • 2. Content Based Image Retrieval (CBIR) 2 Datasets Visual Query Expected outcome: “A dog”
  • 3. Why is a UI useful for CBIR ? ● The importance of visualizing the results ● Ability to capture the user's intent 33
  • 4. Contributions to the project - Development of the UI - Incorporate to the system different modes of interaction - Quantitative and qualitative evaluation 4
  • 5. Contributions to the project - Development of the UI - Incorporate to the system different modes of interaction - Quantitative and qualitative evaluation 5
  • 6. Interconnecting the system DEVELOPING UI 6… choosing tools to develop the project was the next step...
  • 7. ReactJs | NodeJS | Python ● Framework of JavaScript ● Scalability, speed and simplicity. ● Fast ● Virtual DOM ● Fast and scalable network apps ● Single-thread using non-blocking I/O calls ● Capable of handling huge number of simultaneous connections with high throughput. ● NOT able to handle CPU-intensive operations ● Focuses on code readability ● Large standard libraries ● SLOW with speed in request or response petitions processing DEVELOPING UI 7
  • 9. Image datasets DEVELOPING UI ● OXFORD BUILDING (5,063) ● PARIS BUILDING (6,412) ● INSTRE (28,543) Flickr multiple sources 9
  • 11. 11
  • 12. CBIR system ● All images analysed and stored ● Compare query analysed with all the other queries (cosine) ** Mohedano, Eva, Kevin McGuinness, Noel E. O'Connor, Amaia Salvador, Ferran Marques, and Xavier Giro-i-Nieto. "Bags of local convolutional features for scalable instance search." ACM ICMR, 2016. DEVELOPING UI 12
  • 13. INPUTS OF THE SYSTEM URL IMAGE FROM FILE SYSTEM EXAMPLES DEVELOPING UI 13
  • 14. INPUTS OF THE SYSTEM DEVELOPING UI URL SYSTEM EXAMPLES IMAGE FROM FILE 14
  • 15. Contributions to the project - Development of the UI - Incorporate to the system different modes of interaction - Quantitative and qualitative evaluation 15
  • 16. Functionalities of the system ● Explorer mode INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION ● Query expansion mode ● Annotation mode 16
  • 17. Explorer mode : - motivation → Get to know the datasets and explore the system - functioning → When the first query is received and the ranking of the similar images is computed, whichever other image appearing below can be selected now as the new query to search into the dataset. INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION 17
  • 18. Query expansion mode : - motivation → Get to know how the algorithm works in the system. - functioning → Average of the multiple image descriptors selected providing richer representation. INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION 18
  • 19. Annotation mode : - motivation → Improve the accuracy of the automatic system by user’s interaction. - functioning → Annotating images and submit to the system to it can train an SVM. INCORPORATE TO THE SYSTEM DIFFERENT MODES OF INTERACTION 19
  • 20. Contributions to the project - Development of the UI - Incorporate to the system different modes of interaction - Quantitative and qualitative evaluation 20
  • 21. Feedback of users ● UI intuitive ● UI robust and consistent ● UI fully featured ● Understand the purpose of the UI ● Understanding modes with existing explanations ● Explorer mode useful regarding the ‘clickable’ function ● Query expansion mode useful to experiment without affecting the systems accuracy . ● Annotation mode useful to improve accuracy of a trained model. Questionnaire data represented in a graphic Strongly agree Agree QUANTITATIVE AND QUALITATIVE EVALUATION 80% 100% 80% 80% 70% 70% 90% 90% 21
  • 22. Query expansion mode: QUANTITATIVE AND QUALITATIVE EVALUATION ● Low Average Precision → can improve the ranking ● High Average Precision → Just adds noise 22
  • 23. Annotation mode : QUANTITATIVE AND QUALITATIVE EVALUATION ● Possibility to give just the negative feedback ● Possibility to train a model in order of improve the system by just annotating some images of the dataset. 23
  • 25. Conclusions ● UI for an image retrieval system. ● User’s feedback was positive in the questionnaire done. ● UI works with 3 commonly used CBIR benchmarks : ❖ Oxford, Paris and Instre ● Annotation tool has been developed ● Quantitative and qualitative evaluation have been carried out. ● Structure in blocks → Can be adapted for other retrieval algorithms. 25
  • 26. FUTURE WORK ● Include a 'Crop' mode on the query images to specify the region of interest. ● Unify structure of all datasets ● Include a mechanism to measure the time expand per query image. ● Include the ability to search within all photos in the three datasets. 26

Editor's Notes

  1. eina pels cientifics per millorar els seus systems
  2. As I just said, the goal was to develop a UI for a Content Based Image Retrieval search engine system. These systems emerged as a research field in order to fix issues in text-based systems. VISUAL SEARCH They aim of this systems is to structure the datasets based on the content instead of on the metadata associated. They work through algorithms which kind of summarize the content of an image into a numerical vector called ‘image representation’.
  3. It is really important to visualize the results. Most systems, base their evaluation in a quantitative assessment, for example, computing measures as mean Average Precision to perform objectives comparisons with a groundtruth. However, in a real life system there is no groundtruth but just the user. So, qualitative assessments are even more important due the fact that good and bad examples can be shown to understand better how the system works. If there is no groundtruth, visualizing the data can be the only way of testing the results and maybe the only opportunity to realize when a system is failing and correct it. The ability of capture the user’s intent is also really important. This is because it can happen that the similarity concept of the automatic system does not fit with what the user thing is similar. For this reason, it is also really important to develop tools which allows to the user to annotate his perception of whether the results are correct or not. relevance feedback
  4. My contributions on the project can be summarized in 3 fields. The first one would be the development of the user interface the second one would be incorporate to the system different modes of interaction with the users and the last one would be the quantitative and qualitative evaluation.
  5. So, the development of the UI is very important because it will leads us the tool to provide a visualization of the results obtained to, eventually, also improve them by getting the user’s feedback.
  6. To build the system we needed to build first an structure which was able to support all the requirements on the system. We thought about this: We thought about developing a web application, which would be all the client side, and connect it with the image retrival system through a server.
  7. Choosing the tools to develop a project is always a difficult task. However, taking into account the different needs of the project when we are in the designing part can make it easier. 3 languages are presented in the state of art of this project, and they are ReactJS, nodeJS and Python. // EXPLICAR ELS 3
  8. So, taking into account that the server should allow exchanging data with the client in order to provide fast speed and consistent data structures, we decided to built the front end or client part in reactjs and the server in nodejs, due its strength in processing request-response petitions. However, we also needed a server able to support all the mathematical operations in order to calculate the rankings of a query. A solution for this was to build also another server, this one in python, that would incorpore all the image retrieval system code and maintain also a connection with the nodejs server. This connection will be done by ZERO_RPC, which is a library that allows to establish the connection between both programming languages.
  9. The three datasets used are : ** these are very popular datasets in the scientific community working in CBIR. - Oxford Buildings contains 5,063 images and Paris Buildings contains 6,412 images. These both datasets have collected the images from Flickr, a photo sharing community. - In the other hand, we also have Instre dataset. This dataset is larger and contains 28,543 images collected from multiple sources, such as search engines as Google, different social networks as facebook and also photo sharing communities like Flickr.
  10. The aim of the project is to develop an UI capable to compute the list of similar images given a query. So when introducing a query into the system this page will be displayed where you can observe the most similar images at the top of the raking and the less similar ones at the bottom of it.
  11. So, i want to brief a little bit how does the CBIR system works. The system i’m using is from the state of art . it was developed at DCU & UPC and it was awarded in a conference in NY in 2016. In the CBIR system, first all of the images are analysed and stored. Analysing the images follows the next proceedment. first of all, the image is introduced into a pre-trained CNN and decomposed in local deep learning descriptors. Then, this descriptors will be codified by a BoW model using a specific vocabulary. This descriptor vector is the one stored for each image. So, when a query is received into the system, this image is also analysed in order to get the final descriptor vector as gotten above. This vector will be compared with all the other vector’s queries of the system by cosine similarity and then, the ranking will be generated by sorting the similarity scores in descending order. When two images are really similar, the cosine similarity score is nearly 1. Cosine similarity computed Similarity scores in descending order Ranking computed
  12. A web UI is designed so that different alternatives are provided when choosing an image to enter into the system. Thus, the interface will have the option of using some of the already existing images provided as an example by the system, as well as an option of experimenting with a user's own image, either uploading it from a file or providing its URL to the system.
  13. In the second contribution to the project the aim was to incorporate to the system different modes of interaction with the users. Each mode built has different purposes and it is explained in future slides.
  14. The explorer mode’s motivation is to navigate through the system and get to know the different datasets. When a query is received and its ranking is computed, whichever of these images that appear below like in this image, are clickable to become the new query on which one start the new search.
  15. The query expansion mode’s motivation is to check how the deep learning algorithm works by a qualitative evaluation. This mode works by selecting multiples images. The average of their descriptors is done and the search and ranking are computed again. A richer representation is provided to the system, allowing it to improve . In this image it is not very clear but in the demo at the end of the presentation it can be better appreciated. relevance feedback
  16. The annotation’s mode motivation is to improve the system’s accuracy by user’s interaction. Annotate successful and failure images of the ranking and submit the annotations to the system. It is thought to train a model in order to classify what is relevant of what is not. Then, the ranking is update it. support vector machines (SVMs, also support vector networks[1]) are supervised learning models with associated learning algorithms that analyze data used for classificationand regression analysis. functioning → annotate successful and failure images of the ranking and submit the annotations to the system. It is thought to train a model in order to classify what is relevant of what is not. Then, the ranking is update it.
  17. The last contribution is the quantitative and qualitative evaluation of the system. It is done in the first place by the results of a questionnaire, and then through the comparison between query expansion and annotation modes.
  18. An experiment is carried out by asking to 10 people if they can use the system for about 10 minutes and then, answer these questions. The questionnaire was measured with a likert scale where the users choose an option amongst : SD, D, N, A or SA. In this slide it is shown a brief of the results obtained on the questionnaire. The statistics concluded that the UI is intuitive, robust, consistent and built with a useful purpose. The aim of its different modes of operation is understood by the users although some hover info icons are advised in the comments. The users also agreed on the fact that each mode has its own utility thus it is very useful either giving straight feedback to the system as well as being able to experiment with hypothetical situations without affecting it. Last but not least, the fact that an explorer mode is developed with 'clickable' images, has been the most liked feature.
  19. On the QE mode, the 2 next conclusions have been achieved. The first one, is that taking a query with Low Average Precision makes the ranking improves either in the quantitative way as well as in the visualization of the results, what means the qualitative way. The second one, as we can also see in quantitative and qualitative ways, is that when selecting an image with hight AP, it does not improve the accuracy but just introduce noise.
  20. For this reason, if a really accurate ranking wants to be improved, its is much better to use the annotation mode. This is due the possibility of annotating specifically the negative images that doesn’t belong to the ranking . It is also useful the possibility of training an svm because when we have thousands of images we don’t want to annotate the full dataset. Then, instead of annotating all, we can just annotate some of the images and then, train a mode, in this case an svm was trained, which will predict the classification for all the images.
  21. The final conclusions are summarized in this points: So, we built an UI for an image retrieval system and the feedback collect it by the users denoted that the UI is intuitive, robust, consistent and built with a useful purpose. The UI works with 3 commonly used CBIR benchmarks that allow internal and external searches of queries. This are Oxford, Paris and instre dataset. An annotation tool to collect the intent of the users is developed to improve the image retrieval system. Quantitative and qualitative evaluations have been carried out and different scenarios have been studied in order to improve the system The system is designed in a way that the structure can be used for other search engines just changing this module. For this reason, experimenting with another retrieval system could be done and that is why a possible open source contribution has been talked about.
  22. oferta phd submission ACM MM