This document proposes a method to jointly detect humans from on-board and off-board cameras to prevent vehicle-pedestrian accidents. It uses fixed off-board cameras for detection and localization of pedestrians in the environment. On-board cameras on the vehicle are then used to detect pedestrians within detection zones determined from the off-board cameras. An extended Kalman filter fuses the sensor data for improved tracking of pedestrians. Initial experiments show the method works in real-time with improved detection rates compared to only using on-board cameras. Future work involves adding more cameras and GPU processing for improved detection and tracking capabilities.

1. Joint Human Detection from On-Board
and Off-Board Cameras
Justinas Mišeikis and Paulo V. K. Borges

2. Problem
Various vehicles and people share the same environment.
Unfortunately, in these conditions accidents occur.

3. Problem
Can we use technology to
prevent them?

4. Related Work
• Pedestrian detection using vehicle mounted
stereo cameras
• Crowd tracking in enclosed environments
• People tracking using fixed cameras and
driver warning if a person is on the path
• Same object identification using off-board
cameras and not localized mobile camera

5. Our Method Overview
Off-Board camera In world coordinates
Pos relative to On-Board camera
the vehicle detection

6. Our Method Overview

7. Off-Board Cameras
• Fixed cameras
• MOG2 background segmentation
• Noise filtering
• Blob detection
• Size and dimensions filtering
• Feet position - bottom centre
point of the blob

8. People Detector - HOG
• Popular method for people detection
• Works well in cluttered environments
• Based on distribution of intensity gradients
or edge directions.
• Descriptor created from many samples

9. On-Board Cameras
Full Image Analysis
The camera view is split according A: Whole Image analysis - person not
to the distance from the camera: detected
blue (3-7 meters), green (7-12 B: Area split method - person detected
meters), yellow (12-20 meters).

10. Targeted Analysis
Expected feet Detected
position from person
ground plane
homography
HOG search area
Size calculated using the expected person height

11. Data Fusion
Off-Board Cam A Off-Board Cam B
Detected On-Board Cam - HOG
people
positions
Position Fusion
Final Pedestrian Position

12. Data Fusion
Position sensor fusion
6
HOG detector variance
Estimated distance error (meters)
Measurement Error
5
Polynomial Error Estimation
estimation depending 4
on the object’s distance
3
2
from the camera 1
0
2 3 4 5 6 7 8 9 10 11 12
Distance from the camera (meters)
Extended Kalman Filter for sensor fusion and tracking

13. Implementation
• C++
• Standard Intel laptop + desktop running in
parallel
• cvBlob library

14. Experiments
30m by 30m industrial site
Test vehicle - HMC
2 Off-Board, 1 On-Board cam
Three 4-6 minute runs
1-2 People walking around
Vehicle static and moving

15. Results
FP % FN %
Localized Analysis 4.41 3.66
Full Image Analysis 95.53 2.40
• Works real-time
• Background Segmentation - 10-15 FPS
• HOG detectors - 5 FPS

16. Progress
• Additional front facing cameras, 4 in total
- Two side cameras
- Wide angle camera for close proximity
- Far proximity front camera
• People tracking
• HOG performed on GPU GeForce GT640

17. Progress
Close Prox Cam
Danger Zone -
reduced to around
20 cm from the
front of the vehicle
Far Prox Cam

18. Thank You!
Any Questions?