2019/04/24 Intel RealSense Japan Meetup vol.01 Lightning Talk https://tmcn.connpass.com/event/124205 https://youtu.be/ra3KU8rIri8?t=1245

1. OpenCVとPCLでのRealSenseのサポート状況+α

2. Self Introduction
杉浦 司 (Tsukasa Sugiura)
- Freelance Programmer
- Microsoft MVP for Windows Development
- Point Cloud Library Maintainer
- @UnaNancyOwen

3. What is OpenCV/PCL?
Open Source Library for Computer Vision Open Source Library for Point Cloud
- RealSense has been supported on OpenCV 4.0.0
- Input Color, Depth, and Infrared Video Frame
- We have a plan to support RealSense on PCL 1.10.0
- Input Point Cloud with Color, Infrared

4. RealSense Support on OpenCV
- Install RealSense SDK
- Clone OpenCV Source Code from Repository
- Configure and Generate OpenCV Project using CMake
- Build and Install OpenCV
▸ LIBREALSENSE
▹ LIBREALSENSE_INCLUDE_DIR
C:/Program Files (x86)/Intel RealSense SDK 2.0/include
▹ LIBREALSENSE_LIBRARIES
C:/Program Files (x86)/Intel RealSense SDK 2.0/lib/x64/realsense2.lib
▸ WITH
▹ WITH_LIBREALSENSE ☑(check)

5. RealSense Support on OpenCV
#include <opencv2/opencv.hpp>
int main( int argc, char* argv[] )
{
// (1) Open cv::VideoCapture() with RealSense
cv::VideoCapture capture( cv::VideoCaptureAPIs::CAP_REALSENSE ); // Synonym CAP_INTELPERC
if( !capture.isOpened() ){
return -1;
}
while( true ){
// (2) Grab All Frames
capture.grab();
// (3) Retrieve Each Frames
// (3.1) Color Frame
cv::Mat color_frame;
capture.retrieve( color_frame, cv::CAP_INTELPERC_IMAGE );
// (3.2) Depth Frame
cv::Mat depth_frame;
capture.retrieve( depth_frame, cv::CAP_INTELPERC_DEPTH_MAP );
// (3.3) Infrared Frame
cv::Mat infrared_frame;
capture.retrieve( infrared_frame, cv::CAP_INTELPERC_IR_MAP );
// (4) Show Image
cv::imshow( "Color", color_frame );
depth_frame.convertTo( depth_frame, CV_8U, -255.0 / 10000.0, 255.0 ); // Scaling
cv::imshow( "Depth", depth_frame );
cv::imshow( "Infrared", infrared_frame );
const int32_t key = cv::waitKey( 33 );
if( key == 'q' ){
break;
}
}
cv::destroyAllWindows();
return 0;
}
cv::VideoCaptureでRealSenseからデータを取得する | OpenCV Advent Calendar 2018 - 9日目
https://qiita.com/UnaNancyOwen/items/80e8b7c194bf8c334d6d
- Capture Video Frame using cv::VideoCapture()
- Can Handle Same Way as Any Other Camera
- Can’t Detail Settings (Resolution, FPS, ...)
- Can’t Apply Filter of RealSense SDK
- Access Flag Naming ...
UnaNancyOwen/RealSense2Sample | GitHub
https://github.com/UnaNancyOwen/RealSense2Sample

6. RealSense Support on OpenCV

7. RealSense Support on PCL
- Grabber for RealSense is under Development Review
- Build PCL with RealSense SDK from Source Code
- It is required a little patience ...
- Boost, Eigen, FLANN, QHull, VTK, ... Many Dependents
- Build Time is Booooooooooooooooooooooost
PointCloudLibrary/pcl - Add RSSDK 2.0 (librealsense 2.0) grabber implementation #2214 | GitHub
https://github.com/PointCloudLibrary/pcl/pull/2214

8. RealSense Support on PCL
#include <pcl/point_cloud.h>
#include <pcl/point_types.h>
#include <pcl/io/real_sense_2_grabber.h>
#include <pcl/visualization/pcl_visualizer.h>
// Point Type ( pcl::PointXYZ, pcl::PointXYZI, pcl::PointXYZRGB, pcl::PointXYZRGBA )
typedef pcl::PointXYZRGB PointType;
int main( int argc, char* argv[] )
{
// PCL Visualizer
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer
= boost::make_shared<pcl::visualization::PCLVisualizer>( "Point Cloud Viewer" );
// Point Cloud Color Hndler
boost::shared_ptr<pcl::visualization::PointCloudColorHandler<PointType>> color_handler
= boost::make_shared<pcl::visualization::PointCloudColorHandlerRGBField<PointType>>();
// Point Cloud
pcl::PointCloud<PointType>::ConstPtr cloud;
// Retrieved Point Cloud Callback Function
boost::mutex mutex;
boost::function<void( const pcl::PointCloud<PointType>::ConstPtr& )> function =
[&cloud, &mutex]( const pcl::PointCloud<PointType>::ConstPtr& ptr ){
boost::mutex::scoped_lock lock( mutex );
cloud = ptr->makeShared();
};
// RealSense2Grabber
boost::shared_ptr<pcl::RealSense2Grabber> grabber
= boost::make_shared<pcl::RealSense2Grabber>();
// = boost::make_shared<pcl::RealSense2Grabber>( "000000000000" ); // serial number
// = boost::make_shared<pcl::RealSense2Grabber>( "../file.bag" ); // bag file
// Register Callback Function
boost::signals2::connection connection = grabber->registerCallback( function );
// Start Grabber
grabber->start();
while( !viewer->wasStopped() ){
// Update Viewer
viewer->spinOnce();
boost::mutex::scoped_try_lock lock( mutex );
if( lock.owns_lock() && cloud ){
// Update Point Cloud
color_handler->setInputCloud( cloud );
if( !viewer->updatePointCloud( cloud, *color_handler, "cloud" ) ){
viewer->addPointCloud( cloud, *color_handler, "cloud" );
}
}
}
// Stop Grabber
grabber->stop();
// Disconnect Callback Function
if( connection.connected() ){
connection.disconnect();
}
return 0;
}
Drawing Point Cloud retrieve from Intel RealSense Depth Camera (D415/D435) | GitHub Gists
https://gist.github.com/UnaNancyOwen/2ac64d344a27e0c66f802545aefe4ca9

9. RealSense Support on PCL

10. To: Intel RealSense Team
Please Support CMake in Pre-Built RealSense SDK

11. Pre-Built RealSense SDK Not Support CMake
Pre-Built RealSense SDK Self-Built RealSense SDK