Unlike computers, humans process images easier and faster, than text. As users, we become less patient and expect technology to read our minds, and serve the right content. As developers, we have more and more opportunities to work with Machine Learning and Deep Learning. E-commerce companies recognised these trends quickly, introducing image recognition tools, because it brings higher conversion rate. Let’s take a deeper look at visual search: how and where it works, who is using it and how you can do it yourself.

1. Machine Learning
in VISUAL SEARCH
Ola Kunysz
Kraków, 15-17 May 2019

2. Hi,
my name is Ola
I’m a software engineer
I work at Tipser
on shoppable content
and inspirational shopping
OlaQnysz

4. Why (now)?

5. • brain process images faster,
like 60 K faster, than text
• image is international
Humans think
with images

6. • stop words for every language
• translated names and descriptions
• handle bad images
To be
or not
to be:
that is
the
question
:
International search

7. VITA SKOR
Well, it’s related to white shoes indeed…

8. Imperfection
of text search
Do you know what to look for?

9. Is it even “red”? 🤔
Burgundy maybe?
Maybe memory of a sunset?

10. • computers we ready in 80s
• but they’re cheaper now
• and we have cloud
The right hardware

11. No more excuses!
• free start
• pay for what you use
• strong machines for ML

12. Yeah,
but why now?
• $$$
• e-commerce
• inspiration & impatience

13. viewers are more likely to
purchase a product after
watching a product video
of information received
by the brain is visual
of millennials want
visual search over any
other new technology
respondents preferred
visual information
(except electronics and spirits)
90% 85% 62% 50%
WHY YOU SHOULD CARE?

14. Your
workshop

15. TWO
WAYS image to image
image to text

16. 1. Prepare tags/concepts
(or use ready ones)
2. Prepare the training data
3. Train the model
4. Upload the image
5. Extract tags from the image
Image 2 text

17. Fashion-MNIST
• training set of 60,000
• test set of 10,000
• grayscale 28x28 images
• 1 of 10 classes

18. Simple image recognition
link

19. Simple image recognition
link

20. Simple image recognition

21. Simple image recognition

22. diverse results
diverse pricing
similar API/SDK
Third party tools

24. • tag each image
with image recognition
• save tags is search engine (f.e. Elastic)
• tag uploaded image and search for tags
• add context
Use image tags
in text search

25. 1. Use a good model
(or build your own if you have years)
2. Extract features from images
in dataset
3. Upload the image
4. Extract feature from the image
5. Look for the most similar features
Image 2 image

26. Simple Image Search Engine from Yusuke Matsui
https://github.com/matsui528/sis
He uses CNN VGG16
trained on the ImageNet*

27. • dataset of 15 M images
• 1000 categories
• collected by people
• organises challenges
• sets trends

28. VGG16 - the winner of
ImageNet Large Scale Visual Recognition Challenge 2014
Up to 90% of accuracy

30. Example was done on cats (ofc) I did my tests on
from

31. diverse results
diverse pricing
Third party tools

32. Challenges?

33. Images’ quality

34. Model posture

35. Background noise

36. Training data vs real life

37. Dataset relevancy

38. Color detection

39. Dominant color
vs object color

40. Multiple items

41. Color bias

42. • diverse pictures from each
category and subcategory
• grouping by department
• response time
Training data size

43. LET’S RECAP
1. Visual search blooms now and
will get more popular
2. You can do it yourself, use ready
made tools or make a combo
3. You need good quality images
and large training set

44. My personal opinion ;)

45. Want more?
https://yoast.com/visual-search/
https://medium.com/@ODSC/six-big-companies-that-use-visual-search-c98f455057b2
https://blog.logograb.com/evolution-ai-image-recognition/
https://digitalmarketingphilippines.com/visual-search-strategies-to-make-it-work-for-your-business-infographic/
https://medium.com/gsi-technology/ml-in-visual-search-part-i-d54cf4f2b509
https://medium.com/@clarkboyd/visual-search-trends-statistics-tips-and-uses-in-everyday-life-d20084dc4b0a
https://medium.com/syncedreview/visual-search-is-revolutionizing-e-commerce-b27a37dbd296
https://medium.com/@virtua/visual-search-in-e-commerce-41ecf52b66d2
https://elitesem.com/blog/visual-search-ecommerce/
https://medium.com/gsi-technology/visual-search-the-future-of-search-2addfed19689
https://medium.com/nanonets/how-to-classify-fashion-images-easily-using-convnets-81e1e0019ffe
https://hackernoon.com/how-to-build-an-accurate-visual-search-44e04a2830e8
https://medium.com/tensorflow/hello-deep-learning-fashion-mnist-with-keras-50fcff8cd74a
https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/
https://github.com/zalandoresearch/fashion-mnist
https://github.com/matsui528/sis
https://medium.com/tensorflow/hello-deep-learning-fashion-mnist-with-keras-50fcff8cd74a
https://medium.com/nybles/create-your-first-image-recognition-classifier-using-cnn-keras-and-tensorflow-backend-6eaab98d14dd
https://solution44.com/2017/10/04/visual-search-3.html
https://blog.griddynamics.com/reverse-image-search-with-convolutional-neural-networks/
https://www.kdnuggets.com/2019/01/building-image-search-service-from-scratch.html
https://neurohive.io/en/popular-networks/vgg16/
OlaQnysz

46. At the end of a day
it’s all about
multiplying the matrix

47. YOU’RE AWESOME!
OlaQnysz