This is the slides for a paper presented in ICDM workshop in Vancouver-Canada 2011. In the paper we describe a Camshift implementation on mobile robotic system for tracking and pursuing a moving person with a monocular camera. Camshift algorithm uses color distribution information to track moving object. It is computationally efficient for working in real-time applications and robust to image noise. It can deal well with illumination changes, shadows and irregular objects motion (linear/non-linear). We compared the Camshift with a HSV color based tracking and our results show that the Camshift method outperformed the HSV color based tracking. Moreover, the former method is much more robust against different illumination conditions. Paper link: http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6137446&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6137446

1. IMPLEMENTING CAMSHIFT ON A
MOBILE ROBOT FOR
PERSON TRACKING AND PURSUIT
Somar Boubou, Asuki Kouno,
Einoshin Suzuki
Kyushu University, Japan
@LIMER 2011, Vancouver
2011, Dec 11th
1

2. OUTLINE
 Objective and applications
 Problem
 Tracking approaches
 Mean-shift and Camshift
 Related work
 Robot hardware and software
 Experiments
 Results
 Discussion
2

3. OBJECTIVE
A Camshift implementation on mobile robotic
system for tracking and pursuing a moving person
with a camera.
Although researches are very promising, the
problem is still far away from being solved. 3

4.  Museum guidance
[Yamazaki, K- 2009] (CHI
2009)
 Hospital assistance
[Panasonic Medication Dispensing and
Delivery Robots] (2010)
 Socially assistive robot.
[M. Montemerlo 2002]
APPLICATIONS

5. ENVIRONME
NT
ROBOT
University student
office.
Hand made robot equipped with one
camera
• Visual detection and tracking
component.
Detect, track and pursuit.
PROBLEM
Task
5

6. ASSUMPTIO
NS
CONSTRAINTS
DUE TO ON-
BOARD
IMPLEMENTATIO
N
 Tracked person wears two wrist
bands on his both foots.
 Person must be 2m far or less from
the robot, so it can clearly detect
him.
The system has :
• to be robust regarding external
noise (e.g., illumination,
background changes and low-
resolution camera input).
• a sufficient processing speed to
PROBLEM
6

7.  Model-based tracking: Where the geometric structure
of a human body can be represented as 2D or 3D model.
 Region-based tracking: Considers the human body
as a combination of some blobs, representing various body
parts.
 Active-contour-based tracking: Aims at directly
extracting the shape of the subjects by a representation of
the bounding contour of the object and keeps dynamically
updating it.
 Feature-based tracking: Abandons the idea of
tracking objects as a whole, this tracking approach uses
TRACKING
APPROCHES
[Liang Wang 2002]
7

8. MEAN-SHIFT
 Introduced by Y. Cheng. in “Mean Shift, Mode
Seeking, and Clustering” PAMI 1995.
 Mean-shift algorithm
climbs the gradient of a
probability distribution to
find the nearest peak.
8

9. MEAN-SHIFT
Mean shift
vector (follows
density
gradient)
Reference data set
Query point
Center of Mass
x
h
xx
k
h
xx
kx
xm
n
i
i
i
n
i
i








 







 





1
2
2
1
'
'
)(
Mean shift vector with
kernel k
[Dorin Comaniciu 2000]
9

10. MEAN-SHIFT
Reference data set
Query point
Center of Mass
x
h
xx
k
h
xx
kx
xm
n
i
i
i
n
i
i








 







 





1
2
2
1
'
'
)(
Mean shift vector with
kernel k
[Dorin Comaniciu 2000]
Convergence to
cluster center
1

11. CONTINUOUSLY ADAPTIVE MEAN-
SHIFT (CAMSHIFT) [GR Bradski. In1998]
Camshift is used for video sequence,
 CAMSHIFT operates on a probability distribution
image derived from color histograms.
 CAMSHIFT calculates the centroid of the 2D color
probability distribution within its 2D window of
calculation.
 Re-centers the window, then calculates the area for
the next window size.
It differs from Mean-shift: Search window adjusts its
size dynamically.
11

12. FLOW CHART OF PROPPOSED
METHOD
Camshift-based tracking
method
Frame Hue Backprojecti
on
ROIs Tracking
Threshol
dHistogra
m
cf. HSV-based tracking use only threshold to define
1

13. RELATED WORK
 [Fhed 2009]: People Tracking via a Modified Camshift
Algorithm
Fixed platform with static background
 [Simon 2009]: Most Salient Region Tracking
Manual initialization by selecting a rectangle around the
target of interest
 [Feyrer 1999]: Detection, Tracking, and Pursuit of
Humans with an Autonomous Mobile Robot
Used HSI color model combined with motion and contour
information for person detection.
 [Saravanakumar 2011]: Multiple Object Tracking Using
HSV Color Spaces
1

14. OUR WORK
 System Initialization and Detection
Human detection aims at segmenting regions
corresponding to people from the rest of the image.
 Tracking and pursuit
Tracking over time typically involves matching objects in
consecutive frames using features such as points, lines or
blobs.
Originality: Implementing CAMSHIFT for real robot.
1

15. ①SYSTEM INITIALIZATION AND
DETECTION
 Converts the sequence of images into
backprojection images.
 First robot loads the saved
wearable mark template, analyses
it then creates the target
histogram.
 Use mean-shift to define ROIs (Region Of Interest)
we used Camshift to detect the wearable marks
coordinates (u, v).
1

16. α
Go Forward
Turn Right
Turn
Left
𝟓
𝟖
× 𝑼 𝒄𝒐𝒈 − 𝑼 𝒄𝒆𝒏𝒕𝒆𝒓
millisec
Stabilization
range
Action
range
We use the ROI’s windows
defined during detection step to
initialize Camshift instants
used for tracking the wearable
marks.
②TRACKING/ PURSUIT
W : Image width.
FOV : Field of View. FOV
𝑈 𝑎 = α ×
𝑊
𝐹𝑂𝑉
1

17. ROBOT
HARDWARE 6 Touch
sensors
8 IR sensors
USB camera
USB camera
IR sensors
Touch sensors
PandaBoard
Arduino LED unit
Motors
Input MPUs Output
PandaBoard
• 1GHz CPU
• 1GB Memory
• 16 GB SDHC
card
Arduino
• 16 MHz
• 8 KB memory
The two independent components exchange information via a fast inter
process serial communication. 1

18. ROBOT SOFTWARE AND
CONTROLLER PROGRAM
DIAGRAM
The diagram to the right
explains the controller
initialization and
Initialize camera
input
If object
template
new
Load object
template
Find Object/s
Select object
Save object template
Track
object/s
Start
End
Main controlling algorithms, for both
PandaBoard and Arduino board,
are written using C language.
We also implemented
Camshift algorithm,
which is publicly
available from openCV
library.
Ye
s
No
1

19.  Wearable marks (i.e., Wrist bands) with size 6 × 8cm.
 Variables manual calibration of camshift parameters:
Hue (27 < H < 32).
saturation (100 < S < 255).
Value (200 < V < 255).
 10 experiments for each method
where robot pursuit a person
walking inside an office in
approximate path of 11.25 meter
length with average speed of
14cm/sec.
EXPERIMENTAL CONDITIONS
1

20. DESIGN OF THE EXPERIMENTS
We compared Camshift with a HSV color based
tracking methods by evaluating the performance of
the robot in two levels:
 Visual tracking.
 Robot pursuit performance.
Also we compared execution
time for both methods:
• Camshift (363 milisecond)
• HSV (297 millisecond)
Two frames continuous miss-track

21. EXPEREMINTAL RESULTS
 As figure shows Camshift is more robust and stable
since the variance of its results is smaller.
 Temporal distribution of tracking failures affects
pursuit performance
0%
20%
40%
60%
80%
100%
Visual tracking Pursuit
Successrate
Camshift
HSV-based tracking
2

22. DISSCUSSION
 Although, Camshift method is computationally
slower, it outperformed the HSV color based
tracking.
 Camshift method is much more robust against
different illumination conditions.
 Both methods show better tracking results with
larger targets (e.g., tracking the T-shirt or trousers).
2
 During the experiments we realized that the HSV
color based method had serious difficulties in the
first part of our path.

23. PERFORMANCE VEDIO - 2x speed
2

24. Thank you for listening