A really simple presentation about Stereo Vision, especially the stereo vision in real time applied to mobile robotics. In the talk I explained how the stereo vision works, I presented a simplified mathematical model to achieve it and I proposed what for me is the best hardware to use to achieve real-time stereo vision, that is, the sensor ZED produced by Stereolabs used together with the board Jetson TX1 by Nvidia. The presentation is really simple and has been made for an audience with limited knowledge about computer vision and the underhood mathematics. The talk was held during the event "Officine Robotiche 2016" in Rome, 21-22 May 2016

1. Robots that see likeRobots that see like
humanshumans
Real Time Stereo Vision With ZED Camera
on Nvidia Jetson TX1
Walter Lucetti
myzhar@robot-home.it

2. Summary
21/05/2016Officine Robotiche 2016
2
WalterLucetti–myzhar@robot-home.it
Live Demo
Conclusions
Introduction
Stereo “Binocular” Vision
Robotic Stereo Vision
“Simplified” Stereo Vision
Pin Hole Camera model
Stereo Vision camera model
Stereo Vision process
A Robotic Stereo Vision System
Nvidia Jetson TX1
Stereolabs ZED camera
Stereolabs ZED SDK
Stereolabs ZED engine
Simple Code example

3. Introduction3

4. Introduction
4
WalterLucetti–myzhar@robot-home.it
Can a robot replicate the complicated
human binocular vision system?
21/05/2016Officine Robotiche 2016

5. Stereo “Binocular” Vision
5
[from Wikipedia: Binocular vision]
Vision made with TWO “EYES”
It gives a “creature” a spare eye in case one is damaged
It gives a wider field of view
It can give stereopsis in which binocular disparity (or
parallax) provided by the two eyes' different positions on
the head gives precise depth perception
It allows a creature to see more of, or all of, an object
behind an obstacle
It gives binocular summation in which the ability to detect
faint objects is enhanced
WalterLucetti–myzhar@robot-home.it
21/05/2016Officine Robotiche 2016

6. Robotic Stereo Vision
6
How to replicate Human Stereo Vision?
Two synchronized camera sensors
A solid mechanical structure
A powerful elaboration system
and if we want to be more precise, but we want to complicate the
system:
Two motors to replicate vergence capability
WalterLucetti–myzhar@robot-home.it
21/05/2016Officine Robotiche 2016

7. “Simplified” Stereo Vision7

8. “Simplified” Stereo Vision
8
WalterLucetti–myzhar@robot-home.it
A complete Stereo Vision system has
vergence capability, but is too complicated
We will study a Simplified Stereo Vision
system where cameras cannot rotate
21/05/2016Officine Robotiche 2016

9. 9
WalterLucetti–myzhar@robot-home.it
Pin hole camera model
Camera
Plane
Optical
Axis
Image coordinates
World coordinates
World to image
Image to world
Without “Z” we cannot estimate the real dimension of an object using
a single camera!
“f” can be estimated
calibrating the
camera system
This is a simplified
camera model that
does not take care
of
Optical Distorsion
and
Sensor
Imperfections
21/05/2016Officine Robotiche 2016
u=
f ∗x
z
v=
f ∗y
z
x=
u∗z
f
y=
v∗z
f

10. 10
WalterLucetti–myzhar@robot-home.it
Stereo Vision model
Very important parameter:
●
Baseline: b
From triangles similarity:
Very important value:
● Disparity: D = ul
- ur
From disparity we can calculate how
much a point is far from a camera:
THE “Z” VALUE!!!
21/05/2016Officine Robotiche 2016
f
Z P
=
ul
X P
f
Z P
=
−ur
b−X P
ZP=
b∗ f
ul−ur

11. Calculate “Disparity” (D)
for each point of the image
11
WalterLucetti–myzhar@robot-home.it
Stereo Vision process
Calculate the depth (Z) of
each point of the image
Calculate X and Y of each
point of the image
Algorithm
MATCHING
REQUIRES A LOT
OF
COMPUTATIONAL
POWER
D = ul
- ur
21/05/2016Officine Robotiche 2016

12. 12
WalterLucetti–myzhar@robot-home.it
Stereo Vision summary
Allows to calculate real world coordinate
of an object
Requires a solid mechanical structure
Requires a precise calibration of the
optical system
Requires a lot of computational power
…
Is more complicated than what seen until
now
21/05/2016Officine Robotiche 2016

13. A Robotic Stereo Vision system13

14. 14
WalterLucetti–myzhar@robot-home.it
A Robotic Stereo Vision System
Nvidia Jetson TX1
www.nvidia.com/object/jetson-embedded-
systems.html
Stereolabs ZED
www.stereolabs.com
21/05/2016Officine Robotiche 2016

15. 15
WalterLucetti–myzhar@robot-home.it
Nvidia Jetson TK1
Main features
Nvidia Tegra X1 SOC
NVIDIA Maxwell GPU with 256
CUDA cores
NVIDIA 4-Plus-1™ 2.3 Ghz quad-
core ARM® Cortex™-A15 CPU
4 GB LPDDR4 Memory
16 GB eMMC 5.1 Flash Storage
1 full-size SD/MMC connector
1 full-size HDMI port
1 USB 2.0 port, micro AB
1 USB 3.0 port, A
1 RTL8111GS Realtek GigE LAN
GPIO, SPI, I2C, UART, CAN
WiFi & Bluetooth connection
...
 Native OpenCV support with GPU
module for Real Time computer vision
 ROS enabled
 Ubuntu powered
and more...
 Android
 ...
21/05/2016Officine Robotiche 2016

16. 16
WalterLucetti–myzhar@robot-home.it
Stereolabs ZED camera
Main features
Auto calibrating and auto sync
USB 3.0
High Res at high FPS
2208x1242 15 Hz
1920x1080 30 Hz
1280x720 60 Hz
640x480 120 Hz
Simple and well documented C++
SDK
Automatic Depth Map generation
3D Range: 1 m to 20 m
(12 cm baseline)
Outdoor and indoor
Jetson TX1 driver
ROS driver
OpenCV support
21/05/2016Officine Robotiche 2016

17. 17
WalterLucetti–myzhar@robot-home.it
Stereolabs ZED SDK
C++ Classes and APIs for Linux and Windows
2 main classes and 5 support classes
Main classes:
Camera: the main class to use the ZED camera
Mat: the class to store image data in GPU and CPU memory
spaces
Support classes:
CamParameters: calibration params for the two sensors
StereoParameters: calibration params for stereo processing
resolution: dimensions of the images
float3Struct: 3D point
uchar3Struct: RGB pixel
Current version:
0.9.3-beta for Linux and Windows, 0.9.3-alpha for Jetson TX1
21/05/2016Officine Robotiche 2016

18. 18
WalterLucetti–myzhar@robot-home.it
Stereolabs ZED engine
Stereo results are available
in GPU memory
Direct processing by CUDA
or download to CPU memory
for CPU processing
21/05/2016Officine Robotiche 2016

19. 19
WalterLucetti–myzhar@robot-home.it
Stereolabs ZED tools
ZED Explorer ZED Depth Viewer
21/05/2016Officine Robotiche 2016

20. 20
WalterLucetti–myzhar@robot-home.it
Simple code example
//ZED Includes
#include <zed/Camera.hpp>
int main(int argc, char **argv)
{
sl::zed::SENSING_MODE dm_type = sl::zed::RAW;
sl::zed::Camera* zed;
// Camera object creation
zed = new sl::zed::Camera(sl::zed::HD2K);
// We can use different resolutions:
// HD2K HD1080 HD720 VGA
// Sizes of the images
int width = zed->getImageSize().width;
int height = zed->getImageSize().height;
// Initialization of the camera
sl::zed::ERRCODE err =
zed->init( sl::zed::MODE::PERFORMANCE );
if (err != sl::zed::SUCCESS)
{
delete zed;
return 1;
}
// Stereo processing
zed->grab(dm_type);
// CPU Data
sl::zed::Mat depth_cpu;
sl::zed::Mat left_cpu;
sl::zed::Mat right_cpu;
// Data acquiring on CPU memory
depth_cpu =
zed->retrieveMeasure(sl::MEASURE::DEPTH);
left_cpu =
zed->retrieveImage(sl::SIDE::LEFT);
right_cpu =
zed->retrieveImage(sl::SIDE::RIGHT);
// GPU Data
sl::zed::Mat depth_gpu;
sl::zed::Mat left_gpu;
sl::zed::Mat right_gpu;
// Data acquiring on GPU memory
depth_cpu =
zed->retrieveMeasure_gpu(sl::MEASURE::DEPTH);
left_cpu =
zed->retrieveImage_gpu(sl::SIDE::LEFT);
right_cpu =
zed->retrieveImage_gpu(sl::SIDE::RIGHT);
return 0;
}
21/05/2016Officine Robotiche 2016

21. Live demo21

22. Conclusions22

23. 23
WalterLucetti–myzhar@robot-home.it
Conclusions
With Stereo Vision
A robot can see obstacles
A robot can measure obstacles
A robot can localize obstacles
Stereo Vision requires
a lot of mathematical elaborations
a lot of computational power
a solid mechanical structure
so
Nvidia Jetson TX1 with Stereolabs ZED
is a really amazing solution for Robotic Stereo Vision!
21/05/2016Officine Robotiche 2016

24. The End!
OR
THANK YOU FOR YOUR
ATTENTION!
Walter Lucetti
http://myzharbot.robot-home.it
GitHub
@myzhar
Youtube
Myzhar
Twitter
@myzhar
Facebook
robothome
24
21/05/2016Officine Robotiche 2016