Downloaded 29 times
![Stereo “Binocular” Vision
5
[from Wikipedia: Binocular vision]
Vision made with TWO “EYES”
It gives a “creature” a spare eye in case one is damaged
It gives a wider field of view
It can give stereopsis in which binocular disparity (or
parallax) provided by the two eyes' different positions on
the head gives precise depth perception
It allows a creature to see more of, or all of, an object
behind an obstacle
It gives binocular summation in which the ability to detect
faint objects is enhanced
WalterLucetti–myzhar@robot-home.it
21/05/2016Officine Robotiche 2016](https://image.slidesharecdn.com/robotsthatseelikehumans-160524075352/85/Robots-that-see-like-humans-5-320.jpg)
More Related Content
Similar to Robots that see like humans
Robots that see like humans
- 1. Robots that seelikeRobots that see like humanshumans Real Time Stereo Vision With ZED Camera on Nvidia Jetson TX1 Walter Lucetti myzhar@robot-home.it
- 2. Summary 21/05/2016Officine Robotiche 2016 2 WalterLucetti–myzhar@robot-home.it LiveDemo Conclusions Introduction Stereo “Binocular” Vision Robotic Stereo Vision “Simplified” Stereo Vision Pin Hole Camera model Stereo Vision camera model Stereo Vision process A Robotic Stereo Vision System Nvidia Jetson TX1 Stereolabs ZED camera Stereolabs ZED SDK Stereolabs ZED engine Simple Code example
- 3.
- 4. Introduction 4 WalterLucetti–myzhar@robot-home.it Can a robotreplicate the complicated human binocular vision system? 21/05/2016Officine Robotiche 2016
- 5. Stereo “Binocular” Vision 5 [fromWikipedia: Binocular vision] Vision made with TWO “EYES” It gives a “creature” a spare eye in case one is damaged It gives a wider field of view It can give stereopsis in which binocular disparity (or parallax) provided by the two eyes' different positions on the head gives precise depth perception It allows a creature to see more of, or all of, an object behind an obstacle It gives binocular summation in which the ability to detect faint objects is enhanced WalterLucetti–myzhar@robot-home.it 21/05/2016Officine Robotiche 2016
- 6. Robotic Stereo Vision 6 Howto replicate Human Stereo Vision? Two synchronized camera sensors A solid mechanical structure A powerful elaboration system and if we want to be more precise, but we want to complicate the system: Two motors to replicate vergence capability WalterLucetti–myzhar@robot-home.it 21/05/2016Officine Robotiche 2016
- 7.
- 8. “Simplified” Stereo Vision 8 WalterLucetti–myzhar@robot-home.it Acomplete Stereo Vision system has vergence capability, but is too complicated We will study a Simplified Stereo Vision system where cameras cannot rotate 21/05/2016Officine Robotiche 2016
- 9. 9 WalterLucetti–myzhar@robot-home.it Pin hole cameramodel Camera Plane Optical Axis Image coordinates World coordinates World to image Image to world Without “Z” we cannot estimate the real dimension of an object using a single camera! “f” can be estimated calibrating the camera system This is a simplified camera model that does not take care of Optical Distorsion and Sensor Imperfections 21/05/2016Officine Robotiche 2016 u= f ∗x z v= f ∗y z x= u∗z f y= v∗z f
- 10. 10 WalterLucetti–myzhar@robot-home.it Stereo Vision model Veryimportant parameter: ● Baseline: b From triangles similarity: Very important value: ● Disparity: D = ul - ur From disparity we can calculate how much a point is far from a camera: THE “Z” VALUE!!! 21/05/2016Officine Robotiche 2016 f Z P = ul X P f Z P = −ur b−X P ZP= b∗ f ul−ur
- 11. Calculate “Disparity” (D) foreach point of the image 11 WalterLucetti–myzhar@robot-home.it Stereo Vision process Calculate the depth (Z) of each point of the image Calculate X and Y of each point of the image Algorithm MATCHING REQUIRES A LOT OF COMPUTATIONAL POWER D = ul - ur 21/05/2016Officine Robotiche 2016
- 12. 12 WalterLucetti–myzhar@robot-home.it Stereo Vision summary Allowsto calculate real world coordinate of an object Requires a solid mechanical structure Requires a precise calibration of the optical system Requires a lot of computational power … Is more complicated than what seen until now 21/05/2016Officine Robotiche 2016
- 13. A Robotic StereoVision system13
- 14. 14 WalterLucetti–myzhar@robot-home.it A Robotic StereoVision System Nvidia Jetson TX1 www.nvidia.com/object/jetson-embedded- systems.html Stereolabs ZED www.stereolabs.com 21/05/2016Officine Robotiche 2016
- 15. 15 WalterLucetti–myzhar@robot-home.it Nvidia Jetson TK1 Mainfeatures Nvidia Tegra X1 SOC NVIDIA Maxwell GPU with 256 CUDA cores NVIDIA 4-Plus-1™ 2.3 Ghz quad- core ARM® Cortex™-A15 CPU 4 GB LPDDR4 Memory 16 GB eMMC 5.1 Flash Storage 1 full-size SD/MMC connector 1 full-size HDMI port 1 USB 2.0 port, micro AB 1 USB 3.0 port, A 1 RTL8111GS Realtek GigE LAN GPIO, SPI, I2C, UART, CAN WiFi & Bluetooth connection ... Native OpenCV support with GPU module for Real Time computer vision ROS enabled Ubuntu powered and more... Android ... 21/05/2016Officine Robotiche 2016
- 16. 16 WalterLucetti–myzhar@robot-home.it Stereolabs ZED camera Mainfeatures Auto calibrating and auto sync USB 3.0 High Res at high FPS 2208x1242 15 Hz 1920x1080 30 Hz 1280x720 60 Hz 640x480 120 Hz Simple and well documented C++ SDK Automatic Depth Map generation 3D Range: 1 m to 20 m (12 cm baseline) Outdoor and indoor Jetson TX1 driver ROS driver OpenCV support 21/05/2016Officine Robotiche 2016
- 17. 17 WalterLucetti–myzhar@robot-home.it Stereolabs ZED SDK C++Classes and APIs for Linux and Windows 2 main classes and 5 support classes Main classes: Camera: the main class to use the ZED camera Mat: the class to store image data in GPU and CPU memory spaces Support classes: CamParameters: calibration params for the two sensors StereoParameters: calibration params for stereo processing resolution: dimensions of the images float3Struct: 3D point uchar3Struct: RGB pixel Current version: 0.9.3-beta for Linux and Windows, 0.9.3-alpha for Jetson TX1 21/05/2016Officine Robotiche 2016
- 18. 18 WalterLucetti–myzhar@robot-home.it Stereolabs ZED engine Stereoresults are available in GPU memory Direct processing by CUDA or download to CPU memory for CPU processing 21/05/2016Officine Robotiche 2016
- 19. 19 WalterLucetti–myzhar@robot-home.it Stereolabs ZED tools ZEDExplorer ZED Depth Viewer 21/05/2016Officine Robotiche 2016
- 20. 20 WalterLucetti–myzhar@robot-home.it Simple code example //ZEDIncludes #include <zed/Camera.hpp> int main(int argc, char **argv) { sl::zed::SENSING_MODE dm_type = sl::zed::RAW; sl::zed::Camera* zed; // Camera object creation zed = new sl::zed::Camera(sl::zed::HD2K); // We can use different resolutions: // HD2K HD1080 HD720 VGA // Sizes of the images int width = zed->getImageSize().width; int height = zed->getImageSize().height; // Initialization of the camera sl::zed::ERRCODE err = zed->init( sl::zed::MODE::PERFORMANCE ); if (err != sl::zed::SUCCESS) { delete zed; return 1; } // Stereo processing zed->grab(dm_type); // CPU Data sl::zed::Mat depth_cpu; sl::zed::Mat left_cpu; sl::zed::Mat right_cpu; // Data acquiring on CPU memory depth_cpu = zed->retrieveMeasure(sl::MEASURE::DEPTH); left_cpu = zed->retrieveImage(sl::SIDE::LEFT); right_cpu = zed->retrieveImage(sl::SIDE::RIGHT); // GPU Data sl::zed::Mat depth_gpu; sl::zed::Mat left_gpu; sl::zed::Mat right_gpu; // Data acquiring on GPU memory depth_cpu = zed->retrieveMeasure_gpu(sl::MEASURE::DEPTH); left_cpu = zed->retrieveImage_gpu(sl::SIDE::LEFT); right_cpu = zed->retrieveImage_gpu(sl::SIDE::RIGHT); return 0; } 21/05/2016Officine Robotiche 2016
- 21.
- 22.
- 23. 23 WalterLucetti–myzhar@robot-home.it Conclusions With Stereo Vision Arobot can see obstacles A robot can measure obstacles A robot can localize obstacles Stereo Vision requires a lot of mathematical elaborations a lot of computational power a solid mechanical structure so Nvidia Jetson TX1 with Stereolabs ZED is a really amazing solution for Robotic Stereo Vision! 21/05/2016Officine Robotiche 2016
- 24. The End! OR THANK YOUFOR YOUR ATTENTION! Walter Lucetti http://myzharbot.robot-home.it GitHub @myzhar Youtube Myzhar Twitter @myzhar Facebook robothome 24 21/05/2016Officine Robotiche 2016