Yurii Pashchenko: Unlocking the potential of Segment Anything Model (UA) AI & BigData Online Day 2023 Autumn Website – www.aiconf.com.ua Youtube – https://www.youtube.com/startuplviv FB – https://www.facebook.com/aiconf

1. Unlocking the potential of
Segment Anything Model
AI&BigData Online Day 2023

2. About me
❏ Yurii Pashchenko
❏ Principal Machine Learning Engineer at Depositphotos
❏ Over 10 years of research and commercial experience in
applying Deep Learning models
❏ Object Detection/Segmentation and Face Recognition
Specialist

3. Unlocking the potential of Segment Anything Model
● Image segmentation
● Segment Anything Model
● Examples of use
● Limitations

4. Image
Segmentation

5. Basic Image Segmentation
“Image segmentation is a sub-domain of computer vision and digital image processing which aims at
grouping similar regions or segments of an image under their respective class labels”
https://www.v7labs.com/blog/image-segmentation-guide#h3

6. Entity Segmentation
http://luqi.info/Entity_Web/

7. Open-Vocabulary Segmentation
https://paperswithcode.com/task/open-vocabulary-semantic-segmentation

8. SAM: Segment
Anything Model

10. Segment Anything Model (SAM)
SAM: A generalized
approach to
segmentation
https://ai.meta.com/blog/segment-anything-foundation-model-image-segmentation/

11. Segment Anything Model (SAM)

12. SAM.Task
Prompt -> Valid mask
prompt can be
● a set of foreground/ background points
● rough box or mask
● free-form text, or, in general, any information
indicating what to segment in an image.
“valid” mask simply means that even when a
prompt is ambiguous and could refer to multiple
objects the output should be a reasonable mask for
at least one of those objects.

13. SAM.Data Engine
Stage Avg
Annotation
time,s
Avg
Masks/image
Collected
Assisted-manual
stage
34->14 20->44 4.3M masks from
120k images
Semi-automatic
stage
14->34 44 -> 72 5.9M masks in
180k images
Fully automatic
stage
- - 1.1 masks from
11M images

14. SAM.Dataset
SA-1B consists of 11M diverse, high-resolution, privacy protecting images and 1.1B high-quality segmentation
masks that were collected with our data engine.

15. SAM.Model

16. CLIP: Contrastive Language-Image Pre-training
Learning Transferable Visual Models From Natural Language Supervision
● 400 million (image, text) pairs
collected from Internet.
● Trained modifications of
ResNet-50 and ViT-B
● Batch size 32 768 for 32 epochs

17. SAM.Model text encoder

18. Zero-Shot Transfer Experiments
● Single Point Valid Mask Evaluation
● Edge Detection
● Object Proposals
● Instance Segmentation
● Text-to-Mask

19. Examples of use

20. Auto-Label Data with SAM
● Encord
● v7
● Roboflow

21. Matting Anything
https://chrisjuniorli.github.io/project/Matting-Anything/

22. Matting Anything
https://chrisjuniorli.github.io/project/Matting-Anything/

23. Inpaint Anything
https://github.com/geekyutao/Inpaint-Anything

24. Limitations

25. Quality
https://github.com/SysCV/sam-hq

26. Quality
https://github.com/SysCV/sam-hq

27. Meta didn't release the text prompt feature for SAM
Text Encoder.CLIP
CLIP Surgery for Better Explainability with Enhancement in Open-Vocabulary Tasks

28. Meta didn't release the text prompt feature for SAM
Text Encoder.CLIP
https://github.com/xmed-lab/CLIP_Surgery/blob/master/demo.ipynb

29. Text Encoder.DINO
https://github.com/IDEA-Research/Grounded-Segment-Anything

30. Semantic and Panoptic Segmentation
A pipeline for panoptic segmentation can be like this:
1. Use Grounding DINO to detect the "thing" categories (categories with
instances)
2. Get instance segmentation masks for the detected boxes using SAM
3. Use CLIPSeg to obtain rough segmentation masks of the "stuff" categories
4. Sample points in these rough segmentation masks and feed these to SAM
to get fine segmentation masks
5. Combine the background "stuff" masks with the foreground "thing" masks
to obtain a panoptic segmentation label
https://github.com/segments-ai/panoptic-segment-anything

31. Speed.FastSAM
https://github.com/CASIA-IVA-Lab/FastSAM
The Fast Segment Anything Model(FastSAM) is
a CNN SAM trained using only 2% of the SA-1B
dataset published by SAM authors.
FastSAM achieves comparable performance with
the SAM method at 50× higher run-time speed.

32. Speed.Fast SAM
https://github.com/CASIA-IVA-Lab/FastSAM

33. Speed.MobileSAM
https://github.com/ChaoningZhang/MobileSAM

34. Speed.MobileSAM
https://github.com/ChaoningZhang/MobileSAM

35. Thank you for your attention!
AI&BigData Online Day 2023
Yurii Pashchenko
Principal Machine Learning
Engineer at Depositphotos
yurii_pas
george.pashchenko@gmail.com