1. The document proposes developing an intelligent measurement system using machine learning to improve detection rates for defects in steel strips and enable adaptive adjustment of measurement systems. 2. Key aspects include using machine learning models for defect classification and intelligent control of measurement parameters. This aims to increase accuracy and precision while accommodating flexible manufacturing. 3. Measurement data would be uploaded to the cloud and analyzed using intelligent algorithms and big data to provide condition-based maintenance recommendations and predictions to decrease downtime.

1. Copyright 2015 ITRI工業技術研究院機密資料 禁止複製、轉載、外流
ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
智慧檢測技術與工業自動化
量測技術發展中心
儀器與感測技術發展組
周森益
2016.06.16

2. Copyright 2016 ITRI 工業技術研究院
前言
• 面對新經濟模式，自動化生產/檢測，所遭遇
的全新挑戰。
• 當所有資料全部保存到雲端，我們如何看待
這些資料？

3. Copyright 2016 ITRI 工業技術研究院
A Self-Learning Measurement System
with Intelligent Software
Meet industrial
needs:
• In-Situ inspection:
dynamic environment
• Quick response: valid
measurement data and
prediction mechanism
• Flexible manufacture
• User experience data
analysis
1. False alarm
2. Miss detection
3. Unsuitable and manual
coefficients setting from sensors
1. Fault detection decreased
2. Feasible or optimized coefficients settings from sensors
3. Self-diagnosis of maintenance and lifecycle time of sensors
Increase measures’ accuracy & precision
Fault
detection
Sensor 1
Sensor 2
Sensor …
Sensor K
Optical modules
Hardware
modules
Measurement
software
Measurement System
AS IS TO BE
Sensor 1
Sensor 2
Sensor …
Sensor K
Optical modules
Hardware
modules
Measurement System
output
Measurement
software
Intelligent
Software
Self-diagnosis
Self-learning
output
3

4. Copyright 2016 ITRI 工業技術研究院
roller
steel
CCD
Lens
CCD
Lens
AOI system (upper)
AOI system (lower)
Server
User A
User B
Quality control
router
Slitting
process
Top view
Wide steel
Three median
cut steels
Side view
Slitting
process
高速相機
自製光源
待測物
檢測系統規格:
量測範圍: 400mm x133mm
影像解析: 3K x 1K
缺陷檢測解析度: 150 μm
線上檢測速度: 96 m/min
前段執行鋼條切割
與壓延製程
後段執行AOI檢測
瑕疵種類:
刮痕 汙點 鑿痕 油汙
Smart FAB - Top Quality Steel Strip Production Line
應用在消費性產品的鋼帶分條生產線，導入線上全檢目的：
1. 快速診斷不良品問題，可即刻修正提高生產良率。
2. 所有鋼捲皆有檢測資料記錄報告，提供下游客戶更完整的品質保障與投料參考。
4

5. Copyright 2016 ITRI 工業技術研究院
檢測結果軟體頁面
生產線與檢測系統 鋼捲切割分條後產品
1.檢測資料上傳工廠ERP系統，紀錄包含料
號、鋼捲編號、刀序、鋼條寬度等。
2.檢測報表呈現單一鋼捲大範圍檢視，可檢
測出連續型瑕疵，用以改善製程。
面臨挑戰: (產品少量多樣檢測需求)
不同的原物料與製程，鋼捲具有不同的表
面粗糙度與色澤，例如酸洗製程、壓延製
程、鋼板材質與厚度等，將提高缺陷檢出
率的困難度。
5
已建置之生產線上量測系統

6. Copyright 2016 ITRI 工業技術研究院 6
 智能學習架構系統
 higher detection/recognition rate and higher ‎readability report
Training phase
Testing phase
Steel
images
Defect
region
(GO/NG)
Feature
selection
• Area
• Gray value
• ‎Roundness
• …
Training feature
extraction
Training SVM model
Testing feature
extraction
Classifying
Classified
results
Unknown
images
Traditional AOI Learning orientated classification
Feature
correlation &
normalized
AI, machine
learning SVM
Type I:
data points
Type II :
data points
Training model
self learning (linear, non-linear)
Type III :
data points
Unknown pattern
Predict  Type
II
 Quasi-Linear type
 Non Linear type
瑕疵分類SVM

7. Copyright 2016 ITRI 工業技術研究院
Intelligent System
Input layer Hidden layer Output layer
X1
X2
XN
Ɵk f
Ɵk f
Ɵk f
Hk
HNk
Ɵj f
Ɵj f
Y1
Yj
YN0
…
…
…
illumination
Defect features
Steel type/width
Inspection
coefficients
Auto ROI
adjusting
Steel process
Oil Non-oil
PO CR PO CR
Calendering Calendering
8

8. Copyright 2016 ITRI 工業技術研究院
• 提高影像缺陷檢出率之機器學習
- 瑕疵判斷導入機器學習(Machine Learning)機制
提高檢出率(80% ↑)、瑕疵分類製程回饋
• 檢測系統智能化控制與預測
- 檢測系統關鍵參數適應性調整，包括光源亮度、相機參數、處理資料、瑕疵
分類演算法等。
- 系統自我診斷與預測功能，包括光源光衰判斷、系統維護保養、系統組件壽
命偵測等。
因應彈性製造進行智能調整及提升機台稼動率
發展機器學習及決策系統
相機模組
料號不同、材質不同、
系統適應性調整
光學模組
待測物
工廠生產資訊
9

9. Copyright 2016 ITRI 工業技術研究院
Magic T-Box - Windows Phone/Tablet MMI Testing
“Magic T-Box” is a software and hardware integrated solution for Windows tablets and
Windows phones manufacturing functional tests. Industrial Technology Research
Institute and Microsoft collaborate to design the “Magic T-Box” solution to enable
Microsoft OEM and ODM partners to build high quality Windows device efficiently.
1.More than nineteen testing items : System Info, Battery, Brightness, Display, Front
Camera, Rear Camera, Accelerometer, Gyrometer, ALS, Compass, GPS, Touch,
Removable Device, Wifi, Bluetooth, SIM Card, Speaker and mic, Headset and
mic,…,etc.
2.Integrated production record data will be analyzed in the cloud.( 1. Measured
results were upload to database in cloud (Azure). 2. Intelligent algorithm start to
analysis all measured data (Big Data) and show fit results automatically.)
Human Inspection Automatic Magic T-Box
Testing results upload to cloud
10

10. Copyright 2016 ITRI 工業技術研究院
Incoming Quality Control
(IQC)
Raw material input
In Process Quality Control
(IPQC)
Produce (manuf./ assembly)
Out-Going Quality Control
(OQC)
Final Test
TP / Exterior
後相機組裝 前相機組裝
排線整理
絕緣膠片黏貼
機殼外觀檢查 酒精擦拭接觸點
LCD
Others
HDMI
Mic & SpeakerDisplay
Removable
Device
Play Video
Camera
Microsoft’s ODM / OEM (目前現況)
11

11. Copyright 2016 ITRI 工業技術研究院
CCD
Motor MTF Target
Auto Focus
Camera
Resolution
AOI
Color
Temperature
QR Code / Position
AOI : Point / Line / Corner
Chromaticity /
Brightness
SNR
Audio Jack
Holding Fixture
A. Measure Size :
4 ~ 12 inch
B. Adaptive fixture
for all customized
device
ITRI CMS
Magic T-Box
Jitter / Backward /
Input Separation
12

12. Copyright 2016 ITRI 工業技術研究院
Headset
Good qualityPoor quality but acceptable
signal
Sample A Sample B
Sample A：穩定度較差，測試5次內會有1~2次fail的機會
Sample B：穩定度很好，測試都會過
Sample A的音訊以頻譜分析可以發現在極低頻段有明顯的噪
訊，此噪訊不一定會被人耳聽到。Sample B的音訊在頻譜分
析可以觀察到除了訊號外幾乎沒有噪訊。

13. Copyright 2016 ITRI 工業技術研究院
Touchscreen
部分Touchscreen的Report Data有Offset及Jitter的現象
人工測試時會不知不覺地修正Device的錯誤
jitter
Offset
斷線
透過精確地數值分析可確實地過濾不良品

14. Copyright 2016 ITRI 工業技術研究院
19 Testing Items
Cycle Time : < 2 min.
More than 100 output data once
How do you think about it ?
Upload data to the cloud, then do analysis.
+
15
ITRI CMS
Magic T-Box
ITRI CMS Magic Testing Box (T-Box)

15. Copyright 2016 ITRI 工業技術研究院
Fault Detection and Classification
Machine Unstably or Another Product Line ?
Real-time Monitoring
Run to Run Control
Predict
Model
Measured Value Predict Value
Variance
Out of limits
of the recipe?
Update Model
or Data
Clean and
maintain
Adaptive Model for 少量多樣
Predictive Maintenance(PdM)
Maintenance records
Sensor Data
+
Machine Learning
Prediction
Model
input
output
Benefit :
1. decrease downtime event
2. clear what happened
3. predict element’s life time
Not Regular maintenance
User experience independent
16
建立機器本身狀態的自我偵測分析能力

16. 𝑛𝑒𝑡𝑗 = (𝑤1𝑗∗ 𝑥1 + 𝑤2𝑗 ∗ 𝑥2 + 𝑤3𝑗 ∗ 𝑥3) − 𝜃𝑗
17
Prediction Model
𝜔𝑖𝑗
Supervised Learning
Training
Dataset
Input X predicted Y
camera
frame rate
motor
speed
PLC/FPGA
response rate
optical
illumination
air inflow of
pneumatic
cylinder
Maintenance
records
utilization
rate
Predictive Maintenance(PdM)Run by Run Control
… …
𝜃𝑗
𝑛𝑒𝑡𝑗
f
𝑤1𝑗 𝑤𝑖𝑗
𝑥1 𝑥2 𝑥3
𝑤2𝑗 𝑤3𝑗
𝑦1 𝑦2
𝑦𝑗 = 𝑓(𝑛𝑒𝑡 𝑗)
𝑦1
𝜃𝑗
𝑛𝑒𝑡𝑗
f
𝜃𝑗
𝑛𝑒𝑡𝑗
f
𝑥1 𝑥2 𝑥3 𝑥4 𝑥5 𝑥6 𝑥7
Y : dead percentage
Neural
Network
Test Data
1. summation function
2. activity function
3. transfer function
Display Color
Distance
Camera IMF
Sound
Frequency
Indicators

17. SNR : 48.8 dB
Standard Wav.
Time (ms)
Frequency
The Same Input, but output A or B or C.
It’s the sign that 1) DUT Normal 2) DUT Abnormal 3) Diff. DUT.
But how to know in inspect-process ?
A
B
Input Run 2 Run and FDC
C
SNR : 47.7 dB
Diff. DUT

18. Run 2 Run and FDC
43.6 dB
42.4 dB
45.3 dB
41.9 dB
47.2 dB
48.0 dB
46.8 dB 1.5 dB
2.6 dB0.5 dB
1.1 dB
Predict Model
Input
47.6 dB
In Run 2 Run process,
Predict Model will not
be changed and just
modify weight of
Cluster B.
43.6 dB
42.4 dB
45.3 dB
41.9 dB
47.2 dB
48.0 dB
46.8 dB 1.5 dB
2.6 dB0.5 dB
1.1 dB
Predict Model
47.6 dB
Cluster A
Cluster B
Cluster C
Cluster B
Cluster C
Cluster ADUT Normal

19. Run 2 Run and FDC
43.6 dB
42.4 dB
45.3 dB
41.9 dB
47.2 dB
48.0 dB
46.8 dB 1.5 dB
2.6 dB0.5 dB
1.1 dB
Predict Model
Input
37.0 dB
In Run 2 Run process,
Predict Model will be
changed and add New
Cluster D.
43.6 dB
42.4 dB
45.3 dB
41.9 dB
47.2 dB
48.0 dB
46.8 dB 1.5 dB
2.6 dB0.5 dB
1.1 dB
Predict Model
Cluster B37.0 dB
Cluster D
Cluster A
Cluster B
Cluster C
Cluster C
Cluster A
Diff. DUT

20. Run 2 Run and FDC
43.6 dB
42.4 dB
45.3 dB
41.9 dB
47.2 dB
48.0 dB
46.8 dB 1.5 dB
2.6 dB0.5 dB
1.1 dB
Predict Model
Input
2.0 dB
In Run 2 Run process,
Predict Model will not
be changed and know
something wrong.
43.6 dB
42.4 dB
45.3 dB
41.9 dB
47.2 dB
48.0 dB
46.8 dB 1.5 dB
2.6 dB0.5 dB
1.1 dB
Predict Model
Cluster B
Cluster A
Cluster B
Cluster C
Cluster C
Cluster A
2.0 dB
DUT Abnormal

21. SNR : 48.8 dB
Standard Wav.
Time (ms)
Frequency
The Same Input, but sometimes output A, other times output B.
It’s the sign that MIC in mainboard will be broken.
But how to know in inspect-process ?
A
B
Input Example for Predictive Maintenance

22. Example for Predictive Maintenance
Maintenance records
Broken 1 : Normal 5 times, Abnormal 15 times, Diff. DUT 2 times.
Broken 2 : Normal 2 times, Abnormal 7 times, Diff. DUT 1 time.
.
.
.
Broken N : Normal 10 times, Abnormal 27 times, Diff. DUT 4 times.
Input
2.0 dB
Abnormal Abnormal : 6 times
Normal : 2 times
Diff. DUT : 1 time
Broken percentage = 99 %
Predict Model : 𝜔𝑖𝑗
𝑛𝑒𝑡𝑗 = (𝑤1𝑗∗ 𝑥1 + 𝑤2𝑗 ∗ 𝑥2 + 𝑤3𝑗 ∗ 𝑥3) − 𝜃𝑗
𝑦𝑗 = 𝑓(𝑛𝑒𝑡 𝑗)
1. summation function
2. activity function
3. transfer function
𝜃𝑗
𝑛𝑒𝑡𝑗
f
𝑤𝑖𝑗
𝑦1
𝜃𝑗
𝑛𝑒𝑡𝑗
f
𝑥1 𝑥2 𝑥3 𝑥4 𝑥5 𝑥6 𝑥7
Y : dead percentage

23. Copyright 2016 ITRI 工業技術研究院
Combining Machine Learning and Optimization in Supply Chain Analytics
Demand Forecasting
Challenges:
• Predicting demand
for items that have
never been sold
before
• Estimating the
lowest cost
(Price/Time/…).
Techniques:
• Clustering
• Machine Learning
models for
regression.
Cost Optimization
Challenges:
• Structure of demand
forecast
• Demand of each
device is depend on
cost computing ->
exponential number of
variables
Techniques:
• Novel reformulation of
cost optimization
problem.
• Creation of efficient
alg.
1. All combinations were
recorded in cloud(Train)
2. Modeling(Train)
Machine
Learning
Prediction
Model
Expected
Label
建立生產線最佳化的數據分析能力
24

24. Copyright 2016 ITRI 工業技術研究院
未來技術主軸
• 佈建智能化檢測設備
• 檢測設備與生產線的緊密結合
• 上/下游供應商品質資料串連

25. Copyright 2016 ITRI 工業技術研究院
謝謝您的參與和聆聽!
26
感謝技術團隊的支援：
陳心怡 劉曉薇
吳潔瑜 藍于瀅
陳義昌 李漢文
林俊仁 劉定坤
卓嘉弘 王柏愷
陳泳昌