This document describes a multi-camera multi-human tracking system. Local trackers at each camera identify depth order, handle occlusions, and extract body information. The central server compares observations across cameras to determine correspondence, classifies targets, and recognizes people when they reenter by constructing a database. Future work includes expanding to more cameras which increases communication load and complexity, and addressing challenges of outdoor surveillance, uncalibrated systems, and moving cameras.

1. Multi-Camera Multi-Human
Tracking System
Copyright © Yu-Sheng Chen [Yosen]

2. Outline
Top view of the entire tracking system
What local trackers do
Identify people depth order
Handle body occlusion
What central server does
Compare observations across cameras
Classify to different people
Recognize people when re-entering
Yosen Chen, Copyright Reserved 2

3. Images
Local camera
#A tracker
 ,
1
A
tz 2
A
tz
2
B
tz
1
B
tz
Local camera
#B tracker
Images
Central Server
multi-camera correspondence
Person#1 Person#2
Identify how many people
in the observed space
Shared data on Inter-process platform
 ,2
A
tz 1
B
tz 1
A
tz 2
B
tz
Listen/
Broadcast

4. What local trackers do?
Multi-human
tracking &
detection
Central
Server
Operation flow of local trackers
Identify
people depth
order
If detect new,
broadcast
Get people labels
& people heights.
Handle body
occlusion
Start
Keep updating body info. to server databases
Extract
people body
info.
Local tracker
Central server

5. What local trackers do:
 Identify depth order
A. By appearances
B. By standing locations
Image source
1:Front
2:Rear

6. Depth order estimation by head shape
completeness is a bad idea!
Yosen Chen, Copyright Reserved 6

7. What local trackers do:
 Identify depth order
A. By appearances
B. By standing locations
Image source
1:Front
2:Rear

8. Yosen Chen, Copyright Reserved 8
Tell the depth order by people’s
standing locations
How to know the standing locations?
 To be discussed in the central server part!!
Bd Ground plane
Ad
Camera image
Cd
BdAd Cd depth order: A=1, B=2, C=3
A
B
C

9. Depth order estimation by 3D standing
locations is accurate.
Yosen Chen, Copyright Reserved 9

10. What local trackers do:
 Handle body occlusion
 Occluded parts can be estimated by
3D geometry with body modeling

11. Occlusion Estimation by 3D Body
Modeling
Yosen Chen, Copyright Reserved 11
Ground plane
Camera image
B
A
AB
Camera visible parts
Camera invisible parts

12. Body Occlusion Handling
Yosen Chen, Copyright Reserved 12
Depth order
check
Body occlusion
handling
Target info
storage
Body appearance is available for
color extraction
Body appearance is not available

13. What central server does?
Compare
observations
across cameras
Local
trackers
Operation flow of central server
Get observation
correspondence
between cameras
If detect new,
broadcast
Return people labels & people heights.
Start
Keep updating body info. to server database
People
database
Compare with
database to recognize
/create people
Local tracker
Central server

14. What central server does:
 Compare observations across
cameras
 Geometric-based color comparison

15.  2D position on image plane  3D line in real space
 3D standing point in real space  2D feet position on image plane
observation r
Optic center Image plane of
camera j
Ground plane
Optic center
observation q
Image plane of
camera k
3D Geometric Correspondence across
Cameras
Calibrated camera system
Head Point
height
Estimated Standing Location
Yosen Chen, Copyright Reserved 15

16. What central server does:
 Classify to different people
 Use decisions in higher confidence to
determine lower ones.

17. Observation Correspondence
across Cameras
Object Conf = 3
Object Conf = 2
Matching body color
(Block-based body color comparison)
Yosen Chen, Copyright Reserved 17
Different colors mean different labels

18. Multi-People Correspondence
Yosen Chen, Copyright Reserved 18
1. Multi-people depth order √
2. body occlusion handling √
3. Multi-people correspondence √

19. What central server does:
 Identify people if re-enter
 Construct people database once
they got tracked.

20. Yosen Chen, Copyright Reserved 20
Check height
difference
How to recognize people when they re-enter
Are they all from the same person?
Body Color
comparison
by block
Label history
consistency
No, if diff > threshold No, if color unmatched
Yes, they are from
the same person!!
Camera
observations
People model
in database

21. Future Works…
Extend to more cameras
Challenge of
Communication load & algorithm complexity
Tracking/labeling stability
More challenges:
Outdoors Surveillance (human body detection)
Uncalibrated multi-camera system (Training)
Motive cameras (rotation 3D-2D mapping)
Yosen Chen, Copyright Reserved 21

22. Thanks for
Pf. Li-Chen Fu
Advanced Control Lab, NTU
Intelligent Robot Lab, NTU

23. Copyright © Yu-Sheng Chen [Yosen]