AIFrienz_Webinar_Tomomi_Research_Inc).pdf

Tomomi Research Inc.
Trends in AI Visual Inspection
2022-06-15
Tomomi Research, Inc.
Seong-Hun Choe (Dr.Eng.)
https://www.tomomi-research.com/

About Me
2022/6/15 3
At Makers Faire Tokyo 2018
• Name: Seong-Hun Choe （崔成熏)
• Education:
• Seoul National University
BE at Mechanical Engineering
• Tohoku University
Dr. Eng. at Nano Mechanics
• Work at:
• HITACHI, Ltd. (Researcher)
• Hitachi GST -> Western Digital (Principal
Engineer)
• Tomomi Research, Inc. (CTO)
• SNS:
• Twitter : @wireless_power
• Linkedin:
www.linkedin.com/in/seonghunchoe/
• Email: seonghun.choe@tomomi-
research.com
4bit CPU with TTL
TD4
https://makezine.jp/event/make
rs2018/m0268/

Tomomi Research Inc. 2022/6/15 4
Visual Inspection Status in Industrial Area
Labor Shortage
Number of Workers in Visual Inspection:
1.4M in Japan
Ratio in total workers in manufacturing :10~ 20%
Variation in
inspection result
• Depends on individuals
• Human error
• Cost in education

Visual Inspection Status in Industrial Area
単調
熟練、重労働
Variation in
inspection
results
Heavy loads
• Driver Bits
• 10M Pieces/Month
• Failure Rate： 0.01%
• Only 1 or 2 of fails occurs a
day

Market Size
1) Reference: Machine Vision industry Report 2015 Yole Development, Frost&Sullivan analysis
2) Reference: World Bank, U.S. Department of Labor
Automation Rate
Business
Field
100%
Semiconductor Display Electronics Car
Plastic Steel Paper Medicine
Textile Leather Food Airplane
Main Business Filed
Total Market Size
約6.7兆円(2)
Current Market size
7,000億円(1)

Market Mapping
Metal
Food
Electronics
Car
Medicine
Leather
Textile
Plastic
Steel
Semiconductor
Automation
Easy to
see
manual
Hard to
see

1.Classification 2.Object Detection
Four Main Algorithm for Visual Inspection
https://www.youtube.com/watch?v=UY6xbrcViVw&
ab_channel=MobiDev

3.Segementation 4. Anomaly Detection (OK DATA only)
Four Main Algorithm for Visual Inspection
https://www.youtube.com/watch?v=9taNBy7XpFU&
ab_channel=ArayaInc.
Input Output
Anomaly
map
Features can
not be
generated =
Defect

The Higher Yield Rate is, Fewer NG Data will be.
Anomaly Detection with only OK data
• Metal Driver
• 10M Pieces/Month
• Failure Rate： 0.01%
• Only 1 or 2 of fails occurs a
day
Classification Object Detection
Segmentation
Anomaly
Detection with
only OK data

Anomaly Detection on MVTec AD
2022/6/15 11
https://paperswithcode.com/sota/anomaly-detection-on-mvtec-ad

https://www.mvtec.com/company/research/datasets/mvtec-ad
Datasets : MVTec AD

AUROC Two AUROCs on MVTec AD
Metrics: AUROC
Image AUROC

Metrics: AUROC
Pixel AUROC

Metrics: two AUROCs
Image AUROC Pixel AUROC

1. Image Level AD 2. Pixel Level AD
Two Metrics
https://qiita.com/makotoito/items/39bc64d30ce49a
9edad8

Image vs. Pixel AUROC
Image Level AUROC
(Detection AUROC)
Pixel Level AUROC
(Segmentation AUROC)
Definition
• AUROC related to the Image level
• Global Anomaly
• AUROC related the pixel level
• Local Anomaly
Features • Real world Inspection result
• Easy to calculate
• Need to mask image (label image)
• Depends on defect size

Trends in
Anomaly Detection
with only OK Data
18
2022/6/15

Anomaly Detection with Deep Convolutional Autoencoder
1.Autoencoder

Loss function : L^2 (MSE) -> SSIM
1.Autoencoder-SSIM (Structure similarity）
https://arxiv.org/abs/1807.02011

1.Autoencoder-SSIM (Structure similarity）
2022/6/15 21

Pretrained network as a Feature Extractor
2. SPADE： Sub-Image Anomaly Detection with Deep
Pyramid Correspondences [2020/05]
https://arxiv.org/pdf/1112.6209.pdf

Pretrained network as Feature Extractor
2. SPADE：idea
https://static.googleusercontent.com/media/researc
h.google.com/ja//archive/unsupervised_icml2012_s
lides.pdf

• Prequel to SPADE (Same authors)
• Use the pretrained ResNet with ImageNet
data without re-training
• Store the extracted features fy with normal
image data y
• At interference phase, measure the k-
nearest neighbor distance between input
data y and stored feature data fy.
2. SPADE: DN2 Deep Nearest Neighbor Anomaly Detection(2020)

• Store the extracted features f(y,p) with
normal image data y and its each pixel p.
• At interference phase, measure the k-
nearest neighbor distance between input
data y and stored feature data f(y,p).
Image level -> Pixel Level
2: SPADE

Result AUROC
2: SPADE
https://github.com/byungjae89/SPADE-pytorch

Pretrained Feature Extractor + Mahalanobis Distance
3. Gaussian-AD
https://qiita.com/makotoito/items/39bc64d30ce49a
9edad8

Training Inference
3. Gaussian-AD

Result AUROC (Image Level only)
3. Gaussian AD
https://github.com/byungjae89/MahalanobisAD-
pytorch

Pretrained Feature Extractor : Mahalanobis Distance (PaDiM)
• SPADE : slow inference due to kNN
• Mahalanobis Distance instead of kNN
• Store the extracted features with normal
image data y and its each pixel p to the
mean μ and covariant matrix Σ .
• At interference phase, measure the
Mahalanobis distance between input data y
and stored feature data (μ ,Σ ).
4.PaDiM: a Patch Distribution Modeling Framework for Anomaly
Detection and Localization [2020/11]

Result AUROC
4. PaDiM

Pretrained Feature Extractor, Sampling : Coreset Memory Bank.
5.PatchCore: Towards Total Recall in Industrial Anomaly
Detection [2021/06]
• Greedy Coreset to more efficient
sampling from normal data than
random sampling
•Inference time is faster than PaDiM
•SOTA@2022

Result
5.PatchCore

Tomomi Research Inc. 34
Anomaly Detection with MVTecAD
Dataset : MVTecAD
Bottleneck Nuts
Capsule Metal nut

Anomaly Detection with MVTecAD
Dataset : MVTecAD
Carpet Metal grid
Leather Wood

Anomaly Detection History
Generative Model
(~2019)
Pretrained Model as
Feature Extractor
(2019~2022)
Autoencoder
Autoencoder-
SSIM
AnoGAN
DN2 SPADE PaDiM PatchCore
Gaussian AD
kNN
Image
level
kNN
Pixel
level
Mahalanobis distance
Image level
Faster than SPADE
Mahalanobis distance
Pixel level
Faster than PaDiM
Coreset sampling
SSIM
Better result
MSE

Feature Extractor developed by Prof. Otsu Nobuyuki (2011)
6. HLAC (Higher-order Local Auto-Correlation)
Image
HLAC
Feature
Extraction
PCA
Anomaly
Score
Prof. Otsu Nobuyuki
cv2.threshold(gray, 0, 255, cv2.THRESH_OTSU)
https://www.aist.go.jp/pdf/aist_j/syn
thesiology/vol04_02/vol04_02_p70_p
79.pdf

https://adacotech.co.jp/ : Japanse Startup Non-Deep Learning
6. HLAC (Higher-order Local Auto-Correlation)
• Small Train Images : 100~
• No GPU
• Higher Accuracy
https://www.nikkei.com/article/DGXZQOUC09ADH0
Z00C22A2000000/

History of Pattern Recognition
Feature Extractor Design Deep Learning

History of Anomaly Detection
Feature Extractor Design
~2011
Deep Learning
~2019
Pretrained model as
Feature Extractor
2019~
HLAC AutoEncoder
Ano-GAN
SPADE PaDiM
PatchCore
Gaussian AD

Anomaly Detection
in Real Industrial Products
41
2022/6/15

Especially on metal surface (specular surface)
Shim plate
Defects Not Visible Easily

Current Proposed
How to find the defects

Photometric Stereo
Lighting
Position 1
2 4
5 8
3
6 7

Photometric Stereo
Shallow defects
(~10um)
Original Image 2D Texture 3D Surface

Original Image Defects Found!
Invisible defect found!

Original Image Defect Found!
SHIM Plate

https://tomomiresearch.herokuapp.com/
DEMO Site

100 Yen Coin

Strategy
Image Processing
+ =
Deep Anomaly
Detection
Small size AI
Visual Inspection
SystemTR-100

AI Visual Inspection Demo
https://youtu.be/gkk71EKEfxI

Strategy on Market
Metal
Food
Electronics
Car
Medicine
Leather
Textile
Plastic
Steel
Semiconductor
Automation
Easy to
see
manual
Hard to
see

Anomaly Detection Real Industrial Products
Summary
Defects
invisible！

2022/6/15
https://www.tomomi-research.com/ shorturl.at/hzAGL
Please, Subscribe Our YouTube
Channel.

AIFrienz_Webinar_Tomomi_Research_Inc).pdf

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AIFrienz_Webinar_Tomomi_Research_Inc).pdf

Similar to AIFrienz_Webinar_Tomomi_Research_Inc).pdf (20)

More from Seong-Hun Choe

More from Seong-Hun Choe (15)

Recently uploaded

Recently uploaded (20)

AIFrienz_Webinar_Tomomi_Research_Inc).pdf