The document discusses tips and tricks for building automated visual data annotation systems. It covers key topics like what data annotation is, why it is important, challenges in the annotation process, popular computer vision libraries, and techniques like zero-shot classification, few-shot semantic segmentation, and open-vocabulary object detection that can help build automated annotation systems. The document provides examples of how language models like CLIP can be used for tasks like zero-shot classification and relevant image search to help reduce the need for large labeled datasets.

1. Tips and tricks for building your own
automated visual data annotation
systems
Sr ML Engineer at Depositphotos
AI&BigData Online Day 2022

2. About me
❏ Yurii Pashchenko
❏ Sr Machine Learning Engineer at Depositphotos
❏ Over 9 years of research and commercial experience in
applying Deep Learning models
❏ Object Detection and Face Recognition Specialist

3. Tips and tricks for building your own automated
visual data annotation systems
❏ What is data annotation
❏ Types of visual data annotation
❏ Why data annotation is so important
❏ Challenges in the Image Annotation process
❏ Computer Vision Open Source libraries
❏ Annotation of novel classes
❏ Relevant Sample Search
❏ Zero-Shot Classification
❏ Few-Shot Semantic Segmentation
❏ Open-Vocabulary Object Detection

4. What is data annotation
“Computers can't process visual information the way human brains do: A computer
needs to be told what it's interpreting and provided context in order to make decisions.
Data annotation makes those connections.
It's the human-led task of labeling content such as text, audio, images and video so it
can be recognized by machine learning models and used to make predictions.”
What is data annotation and why does it matter?

6. Why it so important
● Data is the backbone of the customer experience. How well you know your
clients directly impacts the quality of their experiences.
● Noticeable correlation between correctly annotated data and the success of
the project
● As much as image annotated data is used to train the machine learning model,
the accuracy will be higher.
● If you are working on an unsupervised machine learning project, sooner or
later, you might need to have data annotation work done if you want to reach
better performance of the algorithms.

7. Challenges in the Image Annotation Process
Balancing costs with accuracy levels: There are two primary data annotation methods—human annotation and
automated annotation. Human annotation typically takes longer and costs more than automated annotation, and also
requires training for annotators, but achieves more accurate results. In comparison, automated annotation is more
cost-effective but it can be difficult to determine the accuracy level of the results.
Guaranteeing consistent data: Machine learning models need a good quality of consistent data to make accurate
predictions. However, data labelers may interpret subjective data differently due to their beliefs, culture, and personal
biases. If data is labeled inconsistently, the results of a machine learning model will also be skewed.
Choosing a suitable annotation tool: There are many image annotation platforms and tools, each providing
different capabilities for different types of annotations. The variety of offerings can make it difficult to choose the
most suitable tools for each project. It can also be challenging to choose the right tool to match the skillsets of your
workforce.
Image Annotation for Computer Vision: A Practical Guide

8. Available annotation tools
1. V7
2. Labelbox
3. Scale AI
4. SuperAnnotate
5. Dataloop
6. Playment
7. Supervise.ly
8. Hive Data
9. CVAT
10. LabelMe
11. LabeIimg
12. VoTT
13. Img Lab
13 Best Image Annotation Tools of 2022

9. Tips and tricks for building your own automated
visual data annotation systems
❏ What is data annotation
❏ Types of visual data annotation
❏ Why data annotation is so important
❏ Challenges in the Image Annotation process
❏ Computer Vision Open Source libraries
❏ Annotation of novel classes
❏ Relevant Sample Search
❏ Zero-Shot Classification
❏ Object Detection/Segmentation
❏ Few-Shot Semantic Segmentation
❏ Open-Vocabulary Object Detection

10. Detectron2
Detectron2
● Facebook AI
● Various tasks
● Unified configs
● Inference speed (fast)

11. OpenMMLab
● Modular Design
● Support of multiple
frameworks out of the box
● High efficiency
● State of the art
OpenMMLab

12. Tips and tricks for building your own automated
visual data annotation systems
❏ What is data annotation
❏ Types of visual data annotation
❏ Why data annotation is so important
❏ Challenges in the Image Annotation process
❏ Computer Vision Open Source libraries
❏ Annotation of novel classes
❏ Relevant Sample Search
❏ Zero-Shot Classification
❏ Object Detection/Segmentation
❏ Few-Shot Semantic Segmentation
❏ Open-Vocabulary Object Detection

13. Relevant image search
Image search engine using Deep Learning Model

14. Relevant image search
Recipe:
1. Preparation
a. Convert images to 224x224x3
b. Extract features by VGG16 model of
size 4096
c. Store embeddings
2. Comparison
a. Extract features for reference image
b. Compute distance (cosine) between
input feature and all embeddings in
dataset
c. Sort distances in ascending order
d. Show first k-images
Image search engine using Deep Learning Model
Limitations:
● only for similarity search
● features are not representative
on real-world samples
● Applicable for datasets 1-10K in
terms of speed

15. Tips and tricks for building your own automated
visual data annotation systems
❏ What is data annotation
❏ Types of visual data annotation
❏ Challenges in the Image Annotation process
❏ Why data annotation is so important
❏ Computer Vision Open Source libraries
❏ Annotation of novel classes
❏ Relevant Sample Search
❏ Zero-Shot Classification
❏ Object Detection/Segmentation
❏ Few-Shot Semantic Segmentation
❏ Open-Vocabulary Object Detection

16. CLIP: Contrastive Language-Image Pre-training
Learning Transferable Visual Models From Natural Language Supervision
● 400 million (image, text) pairs collected
from Internet.
● Trained modifications of ResNet-50
and ViT-B
● Batch size 32 768 for 32 epochs
● The largest ResNet model, RN50x64,
took 18 days to train on 592 V100
GPUs while the largest Vision
Transformer took 12 days on 256
V100 GPUs

17. CLIP for Zero-Shot Classification
Learning Transferable Visual Models From Natural Language Supervision
Ensembling around 80
prompts improve
ImageNet accuracy by
almost 5%

18. Facebook AI Similarity Search (FAISS)
Faiss: A library for efficient similarity search

19. CLIP-based Image Search
Text-to-Image
Unsplash Image Search
"Two dogs playing in the snow"
“The world love written on the wall”
“The feeling when your program
finally works”

20. CLIP-based for Image Search
Image+Text-to-Image
Unsplash Image Search
+
“cars”

21. Tips and tricks for building your own automated
visual data annotation systems
❏ What is data annotation
❏ Types of visual data annotation
❏ Challenges in the Image Annotation process
❏ Why data annotation is so important
❏ Computer Vision Open Source libraries
❏ Annotation of novel classes
❏ Relevant Sample Search
❏ Zero-Shot Classification
❏ Object Detection/Segmentation
❏ Few-Shot Semantic Segmentation
❏ Open-Vocabulary Object Detection

22. You can’t just make an Object Detector
from a Classifier
… without fine-tuning

23. Assembling Object Detector with CLIP
Rich feature hierarchies for accurate object detection and semantic segmentation
CLIP
Text
Encoder
person

24. Region proposals alternatives
Salient Object Detection Techniques in Computer Vision—A Survey
Salient object detection (SOD) is an important computer vision task aimed at precise
detection and segmentation of visually distinctive image regions from the perspective of the
human visual system

25. Region proposals alternatives
Open-World Entity Segmentation
Entity Segmentation is a segmentation task with the aim to segment everything in an image
into semantically-meaningful regions without considering any category labels.

26. Tips and tricks for building your own automated
visual data annotation systems
❏ What is data annotation
❏ Types of visual data annotation
❏ Challenges in the Image Annotation process
❏ Why data annotation is so important
❏ Computer Vision Open Source libraries
❏ Annotation of novel classes
❏ Relevant Sample Search
❏ Zero-Shot Classification
❏ Object Detection/Segmentation
❏ Few-Shot Semantic Segmentation
❏ Open-Vocabulary Object Detection

27. Few-Shot Semantic Segmentation
MSANet: Multi-Similarity and Attention Guidance for Boosting Few-Shot Segmentation

28. Few-Shot Semantic Segmentation
MSANet: Multi-Similarity and Attention Guidance for Boosting Few-Shot Segmentation

29. Tips and tricks for building your own automated
visual data annotation systems
❏ What is data annotation
❏ Types of visual data annotation
❏ Challenges in the Image Annotation process
❏ Why data annotation is so important
❏ Computer Vision Open Source libraries
❏ Annotation of novel classes
❏ Relevant Sample Search
❏ Zero-Shot Classification
❏ Object Detection/Segmentation
❏ Few-Shot Semantic Segmentation
❏ Open-Vocabulary Object Detection

30. Open Vocabulary Object Detection
https://paperswithcode.com/task/open-vocabulary-object-detection/latest
Open-vocabulary detection (OVD) aims to generalize beyond the limited number of base classes labeled during the
training phase. The goal is to detect novel classes defined by an unbounded (open) vocabulary at inference

31. Vision and Language knowledge Distillation
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

32. VILD generalization ability
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

33. VILD visualizations
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

34. VILD visualizations
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

35. Is it worth it?

36. Thank you for your attention!
Yurii Pashchenko AI&BigData Online Day 2022
Yurii Pashchenko
Sr ML Engineer at Depositphotos
yurii_pas
george.pashchenko@gmail.com