Fast and scalable Content-Based Image Retrieval using visual features is required for document analysis, Medical image analysis, etc. in the present age. Convolutional Neural Network (CNN) activations as features achieved their outstanding performance in this area. Deep Convolutional representations using the softmax function in the output layer are also ones among visual features. However, almost all the image retrieval systems hold their index of visual features on main memory in order to high responsiveness, limiting their applicability for big data applications. In this paper, we propose a fast calculation method of cosine similarity with L2 norm indexed in advance on Elasticsearch. We evaluate our approach with ImageNet Dataset and VGG-16 pre-trained model. The evaluation results show the effectiveness and efficiency of our proposed method.

1. A Fast Content-Based
Image Retrieval Method
Using Deep Visual Features
Hiroki Tanioka (Assist. Prof. Ph.D.)
Tokushima University, Japan

2. HELLO!
I am Hiroki Tanioka.
You can find me at @taniokah
2

3. “Quotations are why we
cannot use cosine
similarity on inverted
index-based search
engine, despite cosine
similarity is commonly
used for many
classification problems.”
3

4. Scoring Method
Includes normalizing every
dot product score with L2
norm of each indexed
document.
THERE ARE TWO REASONS FOR THAT
Higher Dimension
Needs much cost of
calculating dot product
between query vector and
indexed document vector.
4
Calculation
Faster
Dimension
Lower

5. Small Ideas
My key concept is using normalized index
and truncating feature vectors.
5

6. 1.
FASTER
COSINE SMILARITY
Let’s start making index with
normalized vectors in advance.

7. COSINE SIMILARITY
TRANSFORM
7
BEFORE:
AFTER:

8. 2.
LOWER
VECTOR DIMENSTION
Let’s reduce features un-preferable
document features for score.

9. SORT AND REDUCE
FEATURE VECTOR
9
・
・
・

10. Dataset Statistics
Number of Images: 25,000
Type: Dogs and Cats
Source: Kaggle
Search Engine
Elasticsearch 7.0.1
INFROMATION OF
EXPERIMENTAL ENVELONMENT
Image Features
Model: VGG-16
Number of Features: 1,000
Score: Probability
Machine Specification
MacBook Pro, 2.5 GHz, 16GB
(Retina, 15-inch, Mid 2015)
10

11. THREE TEST PATERNS
FOR EVALUATION OF
MY SYSTEM
Finding a picture
with original picture
Finding a picture
with low-resolution
Finding a picture
with part of picture
11

12. AVERAGE RESPONSE TIME
Feature 1 2 3 4 5 10 20 30 40 50 100 200 400 1000
dot-prod 0.23 0.33 0.25 0.36 0.37 0.42 0.74 0.69 0.80 0.86 1.43 2.02 2.86 2.87
L1 0.27 0.27 0.30 0.31 0.35 0.35 0.57 0.80 0.70 0.72 1.35 1.58 2.58 2.85
L2 0.26 0.28 0.28 0.30 0.29 0.35 0.51 0.63 0.80 0.99 1.97 3.80 5.78 6.56
cosine 0.23 0.25 0.27 0.28 0.30 0.35 0.64 0.62 0.61 0.59 1.14 1.43 2.60 3.08
dot+cos 44.0 42.2 39.7 36.1 35.0 36.1 35.4 35.0 37.9 41.2 47.0 46.5 52.0 53.0
12
FAST
ENOUGH

13. 13
Comparison MAP of feature number in 100 top-k results and 1.0 resolution rate.
GOOD
ENOUGH

14. 14
Comparison MAP of resolution rates, in 1,000 feature number and 100 top-k
results.
FAIR
ENOUGH

15. 15
Comparison MAP of feature numbers using quartered partial image query, in
1,000 feature number and 100 top-k results.
FAVORABLE

16. Pre-Normalizing
Is effective for calculating
cosine similarity faster
under 100 features.
THERE ARE TWO RESULTS
Reducing Features
Is useful, but at least 8
features are required for our
image search experiment.
16

17. INSTRUCTIONS FOR USE
USE XPACK IN ELASTICSEARCH
https://www.elastic.co/blog/text-
similarity-search-with-vectors-in-
elasticsearch
Elasticsearch 7.3 released
cosine_similarity in xpack.
Still, L1 and L2 are under
construction.
VISIT TO MY GITHUB PAGE
https://github.com/taniokah/icdar-
wml-2019
I’ve just already left my messy
scripts on Github.
I will explain an instruction for my
project.
17

18. 18
THANKS!
Any questions?
You can find me at:
❑ @taniokah
❑ tanioka.hiroki@tokushima-u.ac.jp
This work was supported by JSPS KAKENHI Grant Number JP18H03344.