Abnormal Object Detection under Various Environments Using Self-Organizing Incremental Neural Networks

研究生：黃弘偉
指導教授：范欽雄博士
廖珗洲博士
July 26, 2012
使用自我組織增量神經網路
在各種環境下之異常物體偵測
Abnormal Object Detection under Various
Environments Using Self-Organizing
Incremental Neural Networks

 Introduction
 Moving Object Detection/Tracking
 Abnormal Object Detection Using SOINN
 Experimental Results
 Conclusions
OUTLINE

 Intelligent surveillance
 Labor-intensive jobs are replaced by
machines
 Security, traffic monitoring, crime
prevention…
 Monitoring any abnormal events or
suspicious activities.
 Self-learning
 To detect abnormal objects in different
environments automatically.
 The main functions include moving object
tracking , learning of activity patterns and
abnormal object detection.
INTRODUCTION

 Explicit event recognition
 All events are pre-defined in the knowledge base.
 Modeling of heterogeneous events by labeling them with high-level
semantic descriptors.
 The disadvantages of explicit event recognition
 It is unable to learn an unknown event automatically.
 It is difficult to pre-define all object activities.
 The nature of event varies depending on the environment.
REVIEW OF RELATED WORKS

 Abnormal object detection
 Learning of activity patterns
 activity models are constructed from environment.
 Abnormal object is low frequency activities occurred in the scene.
 Trajectory clustering algorithm
 Hidden Markov models (HMMS)
 Fuzzy self-organized map (FSOM)
 Support Vector Machine (SVM)
REVIEW OF RELATED WORKS (CONT’D)

INTRODUCTION: SYSTEM ARCHITECTURE
Moving Object Tracking Phase
Learning Phase Detection Phase
Object
Profiles
Object
Profiles
Normal
Trajectory
Module
Abnormal
Object
Detection
SOINN
Learning
Abnormality
Results
Object
Profiles
Object
Profiles
Collect
Trajectory
Information
Multi-Object
Tracking Model
Trajectory
PostProcessing
Object Detection
Model
Camera
GMM
Background
Modeling

 Gaussian mixture model
 Every pixel in the image is modeled as the mixture
of k Gaussian distributions.
 The pixel values with high occurrence and low
variation are deemed as the background.
MOVING OBJECT DETECTION/TRACKING :
BACKGROUND MODEL
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Gaussian Mixture Model
𝑤𝑖,𝑡 ∙ 𝜂(𝑥 𝑡; 𝜇𝑖,𝑡, ∑𝑖,𝑡)
𝑘
𝑖=1
 𝑤𝑖 is the respective weight value.
 𝜂(𝑥 𝑡; 𝜇𝑖,𝑡, ∑𝑖,𝑡) is the ith Gaussian
distribution.
 𝜇𝑖,𝑡, ∑𝑖,𝑡 are the mean and
standard deviation, respectively.
 k is the number of Gaussian
distribution

 Gaussian mixture model
 The background model of a pixel (x,y) over the
learning period
 𝑥 𝑡 is the pixel value at t time
 𝑝 𝑥 𝑡 = ∑ 𝑤𝑖,𝑡 ∙ 𝜂(𝑥 𝑡; 𝜇 𝑖,𝑡, ∑ 𝑖,𝑡)𝑘
𝑖=1
BACKGROUND MODEL
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Image Sequence
t
Background

 Parameters update of GMM
 𝑤𝑖,𝑡 = 1 − 𝛼 ∙ 𝑤𝑖,𝑡−1 + 𝛼
 𝜇 𝑖,𝑡 = 1 − 𝜌 ∙ 𝜇 𝑖,𝑡−1 + 𝜌 ∙ 𝑥 𝑡
 𝜎𝑖,𝑡
2
= 1 − 𝜌 ∙ 𝜎𝑖,𝑡−1
2
+ 𝜌 ∙ (𝑥 𝑡−𝜇 𝑖,𝑡)T ∙ (𝑥 𝑡−𝜇 𝑖,𝑡)
BACKGROUND MODEL
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
PixelsPixelsPixels
Is match
k Gaussian
distribution?
Replace
k Gaussian
distribution
Update
k Gaussian
distribution
Yes
No
𝑥 𝑡 − 𝜇𝑖,𝑡−1 ≤ 𝑐 ∙ 𝜎𝑖,𝑡−1

 Target extraction
 Background subtraction method is used to obtain
the foreground image.
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
FOREGROUND DETECTION
𝐹𝑡 𝑥, 𝑦 =
1, 𝑖𝑓 x 𝑡 − 𝜇 𝐵,𝑡−1 > 𝐷 ∙ ∑B,𝑡−1
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

SHADOW REMOVAL
 Shadow characteristic
 A shadow covered a pixel by decreased its
brightness, and the hue value does not change.
 Two information criteria
 brightness distortion
 chromatic distortion
 Morphological operation
 Eliminate some small
fragments.
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking










eotherwis
τ(x,y)B(x,y)I
τ(x,y))B(x,y)(Iβ
(x,y)B
(x,y)I
if α
(x,y)
H
H
k
H
k
S
S
k
S
kV
k
V
k
0
1
Shadow

BLOBS TRACKING
 Blobs (Binary large object)
 Connected component labeling
 Moving object filter
 Noise, non-moving objects, waving trees
 Stable size and speed in successive frames
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
t
t-t+
Blob
Candidate
Moving
Object
Moving
Object

OCCLUSION HANDLING
 Scenarios for tracking in multiple moving
objects
 Non-occlusion phase
 Occlusion phase
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Object-1
Object-2
Object-1
and
Object-2
Object-1
Object-2

OCCLUSION HANDLING
 Mean Shift Algorithm
 Mean shift algorithm climbs the gradient of a
probability distribution to find the nearest domain
mode (peak)
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking

OCCLUSION HANDLING
1. Choose the initial location of the search window.
2. Calculate the PDI of histogram of the object.
3. Use Mean shift algorithm to find the search
window center, and then update the location of the
object.
4. Go to step 3. Repeat the above steps until
convergence
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Tracking

OCCLUSION HANDLING
 Find the centroid of the object in the search window.
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Compute zeroth moment within W:
Find the first moment for x and y:
Compute the centroid within W:
𝑀00 = 𝐼(𝑥, 𝑦)
(𝑥,𝑦)∈𝑊
𝑀10 = 𝑥𝐼 𝑥, 𝑦
𝑥,𝑦 ∈𝑊
, 𝑀01 = 𝑦𝐼 𝑥, 𝑦
𝑥,𝑦 ∈𝑊
𝑥 𝑐, 𝑦𝑐 = (
𝑀10
𝑀00
,
𝑀01
𝑀00
)

OCCLUSION HANDLING
 Disadvantages of mean shift
 Initialize the search window of a target object
manually. Therefore, it is not applicable to the
automated intelligent surveillance system.
 Influenced by time or illumination, histograms of a
target object cannot be updated automatically.
 When the histogram between the target and the
background is similar, the tracking would easily fail.
 When two moving objects with similar histograms
and occlusion occur. Two tracking windows would
follow only one moving object and the other one is
not followed by any tracking window.
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking

 Modified Mean Shift
 More information, such as foreground mask and
moving direction of objects, is added into the back-
projection image(PDI).
OCCLUSION HANDLING
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Tracking

 Steps of Modified Mean Shift
1. Initial location of the search window by making
use of Blobs tracking.
2. Use Kalman filter to predict the location of an
object and set the location as the initial search
window location of mean shift tracking method.
3. Use foreground mask to decrease the influence of
background in the back-projection image of the
object. Use Mean shift algorithm to find the search
window center, and then update the location of the
object.
4. Go to Step 3. Repeat the above steps until
convergence (the search window location moves
less than a preset threshold).
5. Use Kalman filter to correct the search window
location. It can provide a better estimation of
object position.
OCCLUSION HANDLING
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking

 Modified Mean Shift
 Use Kalman filter to correct the search window
location. It can provide a better estimation of object
position.
 It can achieve a stable and accurate mean shift
tracking result.
 It can give more accurate the location and the size
of the search window for mean shift, and solve the
occlusion problem
OCCLUSION HANDLING
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Mean Shift Kalman filter

 Tracking flowchart
HANDLING OF A MISSED TRACKING OBJECT
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Is Occlusion?
Blobs
Tracking
Mean Shift
Kalman Filter
Foreground
Mask
Current
ImageBlobBlobBlobsNew
BlobList
Blobs
Matching
No
Update
Yes
Multi-Object
Tracking Model

 Reason of missed tracking object
 The speed of the moving object is fast
 network transmission delay
 Kalman corrected
 predict the position after 𝜏 frames of an object
 Using velocity predicted the position of a Blob.
 𝑝 𝑘𝑎𝑙𝑚𝑒𝑛 = 𝑝 𝑜𝑟𝑖𝑔 + 𝑉𝑘𝑎𝑙𝑚𝑒𝑛 ∗ 𝜏
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
𝑡𝑡 + 𝜏

 Trajectory extraction
 Feature vectors of object
 𝐹 = (𝑥 𝑡 , 𝑦 𝑡 , 𝑑𝑥 𝑡 , 𝑑𝑦 𝑡 )
 trajectories of object in successive frames
 T= { 𝑥1, 𝑦1, 𝑑𝑥1, 𝑑𝑦1), ⋯ , (𝑥 𝑛, 𝑦 𝑛 , 𝑑𝑥 𝑛, 𝑑𝑦 𝑛 }Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
(x, 𝑦, 𝑑𝑥, 𝑑𝑦) (x, 𝑦, 𝑑𝑥, 𝑑𝑦) (x, 𝑦, 𝑑𝑥, 𝑑𝑦) (x, 𝑦, 𝑑𝑥, 𝑑𝑦)

 Kalman smoothing trajectory
 By the light and shadow in the practical
environment. Therefore, trajectory is often jagged
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking

TRAJECTORY FEATURE EXTRACTION
 Feature extraction
 Position, velocity
 𝐹 = (𝑥 𝑡, 𝑦 𝑡, 𝑑𝑥 𝑡, 𝑑𝑦 𝑡)
 Normalization
 𝐹 = (𝑥 𝑝, 𝑦 𝑝, 𝑥 𝑣, 𝑦 𝑣)
 𝑇 = 𝑥 𝑝,1, 𝑦 𝑝,1, 𝑥 𝑣,1, 𝑦 𝑣,1 , ⋯ , 𝑥 𝑝,𝑚, 𝑦 𝑝,𝑚, 𝑥 𝑣,𝑚, 𝑦 𝑣,𝑚
(𝑚 = 𝑛 ∙ 𝑖)
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking

 Characteristics of SOINN
 Characteristics
 Unsupervised learning method
 Neurons are self-organized with no predefined
network structure and size
 Approximate the topological structure of input data
 Robust to noise
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
input data input data
SOINN: Self−organizing incremental neural network

 Structure of SOINN
 Based on SOM (Self-Organizing Map)
 Two-layer competitive network
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
First Layer
Second Layer
… …Input Layer
 First layer: Competitive
for input data
 Second layer:
Competitive for output of
first layer
 Output topology structure
and weight vector of
second layer

 Algorithm of SOINN
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
𝐴 = 𝑐1, 𝑐2
Initialize:
It has only two nodes.

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Input new pattern
𝜉 ∈ 𝑅 𝑛

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Find the winner and
second winner of input data.
𝑠1 = arg min
𝑐∈𝐴
𝜉 − 𝑊𝑐
𝑠2 = arg min
𝑐∈𝐴{𝑠1}
𝜉 − 𝑊𝑐

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Calculate the
similarity threshold.
𝑇𝑖 =
𝑚𝑎𝑥
𝑗∈𝑁𝑖
𝑊𝑖 − 𝑊𝑗 𝑖𝑓 𝑛𝑜𝑑𝑒 𝑖 ℎ𝑎𝑠 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑛𝑜𝑑𝑒𝑠
𝑚𝑖𝑛
𝑗∈𝐴{𝑖}
𝑊𝑖 − 𝑊𝑗 𝑖𝑓 𝑛𝑜𝑑𝑒 𝑖 ℎ𝑎𝑠 𝑛𝑜 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑛𝑜𝑑𝑒𝑠

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Insert input data
if 𝜉 − 𝑊𝑠1
> 𝑇𝑠1
or 𝜉 − 𝑊𝑠2
> 𝑇𝑠2
,
then 𝐴 = 𝐴 ∪ 𝑟 and 𝑊𝑟 = 𝜉.

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Input new pattern
Calculate the

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Connection between winner
and second winner.
𝐶 = 𝐶 ∪ (𝑠1, 𝑠2)

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
It has this structure

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Find the winner and
second winner of input data.

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Calculate the

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Update the weight vector of
winner and its neighbors.
∆𝑊𝑠1
= 𝜖1 𝜉 − 𝑊𝑠1
∆𝑊𝑖 = 𝜖2(𝜉 − 𝑊𝑖)(∀𝑖 ∈ 𝑁𝑠1
)
𝜖1 = 1/𝑀𝑠1
、𝜖2 = 1/100𝑀𝑖.

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
It has this structure.

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Find the nodes whose
neighbor is less than or equal to 1.
if 𝐿𝑖 = 0 ∀𝑖 ∈ 𝐴 or 𝐿𝑖 = 1 ∀𝑖 ∈ 𝐴 ,
then 𝐴 = 𝐴{𝑖}

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Delete such nodes.

 Normal trajectory module
 Trajectory information of objects are collected.
 When trajectories are inputted, SOINN is used to
construct a normal trajectory module.
 The module is then used to analyze moving objects
in the real-time camera frame and find out
abnormal objects.
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Normal Trajectory Module
Position Velocity

 Abnormal object detection
 Decide whether an observed object is abnormal or
not .
 For ith object trajectory
𝑇𝑖 = { 𝑥1, 𝑦1, 𝑑𝑥1, 𝑑𝑦1), ⋯ , (𝑥 𝑛, 𝑦 𝑛, 𝑑𝑥 𝑛, 𝑑𝑦 𝑛 }
 𝑇𝑖 is matched to the normal trajectory module.
Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
Normal Trajectory Module
Position Velocity

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
𝐷 𝐹 = 𝑊𝐹 − 𝑊𝑠
𝑇s = max 𝑇𝑠1
, 𝑇𝑠2
𝑠 = arg max(𝑇𝑠1
, 𝑇𝑠2
)

Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
𝐷𝑠𝑢𝑚 = 𝐷 𝐹,𝑖
𝑛
𝑖=1
, 𝑇𝑠𝑢𝑚 = 𝑇𝑠,𝑖
𝑛
𝑖=1

 In the real world, definition of abnormality is a
fuzzy concept. In addition, the occurrence of
abnormal objects is continuous, which is not
discrete.Object detection
Current image
Anomaly detection
Learning trajectory
Occlusion Handling
Object tracking
𝑅d = 𝐷𝑠𝑢𝑚 − 𝑇𝑠𝑢𝑚 𝐷𝑠𝑢𝑚 + 𝑇𝑠𝑢𝑚
𝐶 = 𝐶 +
1, 𝑖𝑓 𝑅 𝑑 > 𝑅 𝑇
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝐴𝑂 =
𝑡𝑟𝑢𝑒, 𝑖𝑓 𝐶 ≥ 𝐶 𝑇
𝑓𝑎𝑙𝑠𝑒, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝑅 𝑇 from 0.6 to 0.8

EXPERIMENTAL RESULTS :
SYSTEM INTERFACE
Main
screen
Anomaly
detection
button
靈敏度
Toolbar
Moving
objects

 Foreground detection
 Including the GMM,
shadow removal method,
morphological operation
 Avg. execution time:
0.025 – 0.030 sec / per
image(320x240)
MOVING OBJECT TRACKING

 Multi-objects tracking
 Including the Blobs tracking,
modified mean shift
 In the occlusion case, each
occluded object takes extra
0.002 – 0.003 sec to handle
occlusion.
 Avg. execution time: 0.004 –
0.018 sec / per
image(320x240)

 Comparison of Scenarios characteristics
Scenarios
Trajectory
complexity
Speed of
objects
Occlusion
frequency
School campus High Slow Medium
Roads Medium Fast High
One-Way Street Low Very fast Medium

 Tracking successful rate:
 classified as successful and failed in different resolution
 Successful: tracked successfully for more than 90% of the ground
truth
 Failed: tracked successfully for less than 90% of the ground truth
 Avg. successful rate: 96.9%
Scenarios
Number
of objects
Successful rate (%)
𝟑𝟐𝟎 ∗ 𝟐𝟒𝟎 𝟐𝟒𝟎 ∗ 𝟏𝟖𝟎 𝟏𝟔𝟎 ∗ 𝟏𝟐𝟎
School campus 124 99.1% 100% 99.1%
Roads 158 96.8% 96.2% 96.2%
One-Way Street 88 94.3% 95.5% 95.5%

 Reason of tracking error
 Between object and background color is similar
 Object size is over large(佔鏡頭畫面過大)
 Object speed is over fast

 Trajectory date: 2350
 SOINN neuron: 367
 Avg. learning time: 41 – 43 sec
ABNORMAL OBJECT DETECTION

 Performance of Abnormal object detection
 ACC is the ratio of the sum of all detected object is truly for all
objects.
 RC is the ratio of the probability that the abnormal object are
detected.
Seconds
True
Positives
True
Negatives
False
Positives
False
Negatives
Accuracy
(ACC)
Recall
(RC)
School campus 13 137 0 0 100.0% 100.0%
Roads 10 231 3 1 98.3% 90.9%
One-Way Street 18 63 1 0 98.8% 100%
𝐴𝐶𝐶 =
𝑇𝑃 + 𝑇𝑁
𝑇𝑃 + 𝐹𝑁 + 𝐹𝑃 + 𝑇𝑁
, 𝑅𝐶 =
𝑇𝑃
𝑇𝑃 + 𝐹𝑁

 Reason of misdetection
 Most of the misdetection cases are caused by tracking error. Then,
the wrong trajectory information is mismatched with the normal
trajectory model.
 If there are too many trajectory instances for the SOINN to learn, the
complexity is high and it may cause over-fitting problem.

 The proposed method can solve the problem of object
occlusion. Effective the extract object trajectories, and reduce
their noise.
 The proposed method is a self-organizing method to learn
trajectories for abnormal object detection.
 Abnormal object detection under 3 different scenarios.
 Campus squares, roads and one-way street.
 Avg. accuracy is 99% and recall is 96.7%.
 Avg. execution time is from 33 to 67 milliseconds(real-time).
CONCLUSIONS

 More good background model and shadow removal method
 To detection under different weather conditions.
 Occlusion handling
 Using V. Papadourakis, A. Argyros and GMM in object modeling can
be also refined to improve the tracking accuracy.
 Learning method
 A faster and more efficient
 Using S. Furao, T. Ogura and O. Hasegawa to reinforce the original
SOINN.
 Abnormal threshold
 More accurate abnormal threshold value
FUTURE WORKS

 Application
FUTURE WORKS
!

THANK YOU FOR LISTENING
感謝各位口試委員的蒞臨與指導！

Abnormal Object Detection under Various Environments Using Self-Organizing Incremental Neural Networks

Recommended

Recommended

More Related Content

Similar to Abnormal Object Detection under Various Environments Using Self-Organizing Incremental Neural Networks

Similar to Abnormal Object Detection under Various Environments Using Self-Organizing Incremental Neural Networks (20)

Recently uploaded

Recently uploaded (20)

Abnormal Object Detection under Various Environments Using Self-Organizing Incremental Neural Networks