Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"Building Complete Embedded Vision Systems on Linux—From Camera to Display," a Presentation from Montgomery One

121 views

Published on

For the full video of this presentation, please visit:
https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2019-embedded-vision-summit-montgomery

For more information about embedded vision, please visit:
http://www.embedded-vision.com

Clay D. Montgomery, Freelance Embedded Multimedia Developer at Montgomery One, presents the "Building Complete Embedded Vision Systems on Linux—From Camera to Display" tutorial at the May 2019 Embedded Vision Summit.

There’s a huge wealth of open-source software components available today for embedding vision on the latest SoCs from suppliers such as NXP, Broadcom, TI and NVIDIA, at lower power and cost points than ever before. Testing vision algorithms is the first step, but what about the rest of your system? In this talk, Montgomery considers the best open-source components available today and explains how to select and integrate them to build complete video pipelines on Linux—from camera to display—while maximizing performance.

Montgomery examines and compares popular open-source libraries for vision, including Yocto, ffmpeg, gstreamer, V4L2, OpenCV, OpenVX, OpenCL and OpenGL. Which components do you need and why? He also summarizes the steps required to build and test complete video pipelines, common integration problems to avoid and how to work around issues to get the best performance possible on embedded systems.

Published in: Technology
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

"Building Complete Embedded Vision Systems on Linux—From Camera to Display," a Presentation from Montgomery One

  1. 1. © 2019 Montgomery One Building Complete Embedded Vision Systems on Linux - Camera to Display Clay D. Montgomery Montgomery One May 2019
  2. 2. © 2019 Montgomery One The Big Picture - Overview • Start on a Ubuntu Linux PC to develop and prototype new CV algorithms and applications - Install, build and explore the OpenCV sample programs on a Ubuntu system • Select your Open-Hardware ARM SoC (System on Chip) board and cameras carefully - A good BSP with camera drivers and Yocto support are critical! • Build a custom OS with Yocto Open Embedded Linux for your target board - Use the same open-source libraries on Ubuntu and target systems, if at all possible • Build your application with the Yocto toolchain for your ARM target system • Select Open-Source component libraries to use - Which do you actually need and which are the best for your application? - V4L2, FFmpeg, GStreamer, OpenCV, OpenGL ES, OpenCL and OpenVX? • Explore the options for acceleration on your ARM SoC - Acceleration is required for most CV applications, so plan for it 2
  3. 3. © 2019 Montgomery One Popular ARM SoC Boards for Vision 3 RaspberryPi 3 TI BeagleBoard NXP i.MX8 Nvidia Jetson TX2 i.MX6 WandBoard with MIPI Camera
  4. 4. © 2019 Montgomery One Exploring Open Source Video Components for Linux
  5. 5. © 2019 Montgomery One An Example - Vision Components for Lightwing 5
  6. 6. © 2019 Montgomery One V4L2 – Video for Linux (Version 2) 6 • Negotiates compatible video formats and manages buffers and events between apps and drivers • The de-facto standard low-level video API for Linux • The most widely used API for video input on Linux systems • Most Linux distros and BSPs provide drivers for cameras and TV tuners for V4L2 • Version 1 is obsolete, but still used by the OpenCV VideoCapture class • Well supported by most SoC vendors that have integrated MIPI camera support • The UVC standard (USB Video Class for web cams) is newer and still uses V4L2 on Linux
  7. 7. © 2019 Montgomery One GStreamer • Negotiates compatible video formats and builds pipelines and filter-graphs to connect plug-in components and apps to the V4L2 API • The de-facto standard high-level audio/video API for Linux • The most widely used API for video pipelines on Linux systems • Used by OpenCV's VideoCapture class to work around video format issues, such as color space and RGB/BGR mismatch issues gst-launch-1.0 autovideosrc ! videoconvert ! autovideosink ./videocapture_basic "imxv4l2videosrc device=/dev/video0 ! videoconvert ! appsink" • Well supported by some SoC vendors that have integrated VPUs and codecs - VPUs can be very difficult to exploit without good drivers for GStreamer - This is a major differentiator among SoC vendors and should be evaluated carefully 7
  8. 8. © 2019 Montgomery One OpenCV • 2D and 3D feature toolkits • Egomotion estimation • Facial recognition system • Gesture recognition • Human–computer interaction (HCI) • Motion tracking and understanding • Object segmentation and recognition • Depth perception from 2 cameras • Structure from motion (SFM) • Statistical machine learning and DNN • Support for OpenCL and CUDA accelerators 8 • Not a Khronos standard, but mostly open-source • Began as Intel Research open initiative in 1999 • Intel donated OpenCV to non-profit OpenCV.org • Most comprehensive and widely used CV library • Available on most platforms and languages today • Lots of sample programs in C/C++ and Python • Great place to start to prototype a new design • Latest 4.x available, but not much help on ARM FeaturesOrigin and Status Issues • Typically too slow for camera video (on ARM) • Samples require X11 on Linux • Bugs in VideoCapture class on Linux • Requires video in BGR format
  9. 9. © 2019 Montgomery One FFmpeg • A set of libraries for encoding, decoding and converting audio and video files and streams, such as JPEG, MPEG and H.264, etc. • Supported on a very wide range of platforms • Very limited support for accelerators on ARM SoCs (NEON) - Most ARM SoC vendors do not provide drivers to accelerate FFmpeg - FFmpeg codecs are typically too slow for camera video on ARM SoCs - But, still useful for de-muxing, removing containers, etc. 9
  10. 10. © 2019 Montgomery One OpenGL ES • The original Khronos consortium standard for 3D graphics on SoCs • One of the most successful and widely adopted APIs for embedded multimedia ever • Version 1.1 is obsolete • Versions 2.0 - 3.0 are widely supported today by most SoC vendors with GPUs • Shader code can be used to accelerate some CV algorithms • Version 3.1 added more abilities to do general-purpose compute, including CV • Driver support is more mature and stable than for OpenCL and OpenVX • GLSL coding is familiar to more developers than OpenCL, NEON or CUDA • Lightwing uses version 2.0 GLSL for faster motion tracking than is possible with OpenCV 10
  11. 11. © 2019 Montgomery One OpenCL 11 • Khronos consortium standard for general-purpose compute acceleration on SoCs and FPGAs • Framework for general-purpose compute acceleration across heterogeneous processors including CPUs, GPUs, DSPs and FPGAs • Based on C99 and C++11 languages • Provides a standard interface for parallel computing using task- and data-based parallelism • Some mid to high-end SoC and FPGA vendors support OpenCL for CV applications • Open alternative to Nvidia's CUDA • Not as mature on most platforms as OpenGL ES or CUDA • Will accommodate a wider range of algorithms than OpenGL ES shader code
  12. 12. © 2019 Montgomery One OpenVX 12 • Khronos standard for cross-platform acceleration of computer vision applications • A higher level of abstraction for programming CV than OpenCL • Based on a connected graph of vision nodes that execute a preferred chain of operations • Complementary to OpenCV, but can offer more optimized graph management • Many SoC vendors are quickly developing support for OpenVX • Available now on NVIDIA • Supports face, body and gesture tracking, smart video surveillance, advanced driver assistance systems (ADAS), object and scene reconstruction, augmented reality, visual inspection, robotics, etc.
  13. 13. © 2019 Montgomery One Basic Camera Video Pipelines 13
  14. 14. © 2019 Montgomery One Your First Camera Video Pipeline (V4L2 and GStreamer) • Attach a USB Camera to a Ubuntu PC (or your target system) and try: dmesg | grep camera - Displays kernel error messages about your camera driver's initialization ls /dev/video0 cat /dev/video0 - Shows if your camera driver is installed and working (producing data) lsmod - Lists all installed kernel drivers gst-launch-1.0 videotestsrc ! autovideosink - Test to see if gstreamer is installed and working (should display a color bars test pattern) gst-launch-1.0 autovideosrc ! autovideosink - Initialize and run a complete video pipeline (camera to display) using V4L2 and gstreamer 14
  15. 15. © 2019 Montgomery One OpenCV Sample Programs
  16. 16. © 2019 Montgomery One Explore the OpenCV Sample Programs • Install OpenCV 3.2 on Ubuntu (14 - 16) and build the sample apps: sudo apt-get install opencv sudo apt-get install build-essential sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev cd ~/opencv-3.2.0/opencv mkdir release cd release cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. make sudo make install cp -r /usr/local/lib/libopencv* /usr/lib/. • Run some sample apps on Ubuntu: cd /usr/share/OpenCV/samples/bin ./cpp-example-videocapture_basic • Combine sample source codes with 'videocapture_starter' to test an algorithm with camera video - cpp-example-videocapture_starter.cpp + cpp-example-edge.cpp 16
  17. 17. © 2019 Montgomery One Yocto Open Embedded Linux
  18. 18. © 2019 Montgomery One Build a Yocto Linux System for Your Target Board • Choose a board with a Yocto BSP and cameras with recipes for Yocto • Test the camera drivers provided in your BSP well - You will likely have to create recipes and/or a new V4L2 driver for your camera - An example of a working driver from your camera vendor is a minimum requirement • Install Yocto (Pyro, Rocko or Sumo) on Ubuntu and build a bootable OS: repo init -u https://github.com/Freescale/fsl-community-bsp-platform -b pyro repo sync MACHINE=wandboard DISTRO=fslc-x11 source setup-environment build bitbake fsl-image-machine-test-full (or, core-image-basic, fsl-image-x11-qt5, etc.) • Build and install cross-development toolchain to build your app and any additional packages: bitbake fsl-image-machine-test-full -c populate_sdk ./build/tmp/deploy/sdk/fslc-x11-glibc-x86_64-fsl-image-multimedia-armv7at2hf-neon-toolchain-2.3.4.sh bitbake -s 18
  19. 19. © 2019 Montgomery One Integrating OpenCV with Yocto Linux
  20. 20. © 2019 Montgomery One An Example - Vision Components for Lightwing 20
  21. 21. © 2019 Montgomery One Build and Run OpenCV Samples for Your Target Board • Add packages for OpenCV, X11, gstreamer, etc. to Yocto's config file: buildconflocal.conf CORE_IMAGE_EXTRA_INSTALL += "opencv x11 gstreamer" • Build and test the OpenCV 3.2 samples using Yocto toolchain, sysroot and CMake in your Yocto tree (mostly the same as on Ubuntu): source /opt/fslc-x11/2.3.4/environment-setup-arm7at2hf-neon-fslc-linux-gnueabi cd release cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. make • Running the sample apps on your target requires initialization of an X window (X11): export DISPLAY=:0.0 xhost +SI:localuser:root cd /usr/share/OpenCV/samples/bin ./cpp-example-videocapture_basic 21
  22. 22. © 2019 Montgomery One VideoCapture Class for Linux • The camera interface for OpenCV • About 40 implementations are provided for all major OS platforms • Three implementations are provided for Linux: cap_v4l.cpp - Original, for obsolete V4L version 1 cap_libv4l.cpp – Updated replacement for V4L2, with many patches cap_gstreamer.cpp – Optional extension convert formats and accepts GStreamer pipeline syntax: cpp-example-videocapture_basic 'videotestsrc ! videoconvert ! appsink' • Required some debugging to work on newer (4.x) Linux kernels - If V4L2 fails to initialize, it then tries V4L (version 1), which produces very confusing error messages. - Removed the xioctl() call macro because it was failing when calling to inquire the video format (VIDIOC_S_FMT) • Some improvements are available in OpenCV 4.1.0 for the Android Media NDK 22
  23. 23. © 2019 Montgomery One What Else Can (and Did) Go Wrong? • OmniVision 5640 camera discontinued –Was a popular camera supported by the FSL Community BSP –Use new OV5645 driver from Rocko version instead –Only older versions of Yocto (Pyro) actually work on most i.MX6/8 boards • Yocto only supports OpenCV 3.2 because of CMake compatibility issues –CMake installed by Yocto will not build OpenCV 3.4 • OpenCV samples require X Windows on Linux –Either use X11 or eliminate the use of the High-GUI module in samples • OpenCV VideoCapture class requires GStreamer on Linux –Unnecessarily complex and broken on the i.MX6/8 platform –Fixed the bugs in V4L2 implementation, so it works without GStreamer 23
  24. 24. © 2019 Montgomery One Working Around Component Limitations • OpenCV (VideoCapture class) requires the antiquated 24 bit BGR pixel format ! –Many cameras only provide 16 bit RGB 565, YUV or RAW formats –RAW format is 10 bit RGB (Bayer), but requires software conversion (very slow) • Official solution is to use gstreamer (with an appsink) element to convert YUV to BGR: videocapture_starter 'imxv4l2videosrc ! videoconvert ! appsink' - Can be accelerated by the IPU on i.MX6/8 by using NXP's plugins for gstreamer - libgstapp.so must be installed manually for this to work due to a bug in some builds for i.MX6/8 • Other ways to adapt the V4L2 VideoCapture class and get better performance: –Use the i.MX6/8 IPU without GStreamer. Red and Blue are swapped, but for many CV apps, that is fine –Use luminance data directly (dropping color data), without conversion to RGB • i.MX6 IPU will not scale video beyond 1024 x 1024 pixels (fixed on the i.MX8) • Lightwing solution - Use the GPU (OpenGL ES) to convert and scale video, instead of the IPU or GStreamer - Basic motion tracking algorithm is done in shader code, instead of OpenCV 24
  25. 25. © 2019 Montgomery One Lightwing Motion Tracking Demo
  26. 26. © 2019 Montgomery One Lightwing Motion Tracking 3D GUI Demo Motion tracking on the NXP i.MX6 GPU enables interactive control of 3D assets for mixed-reality video walls, without VR headsets or hand controllers 26
  27. 27. © 2019 Montgomery One Conclusions
  28. 28. © 2019 Montgomery One Lessons Learned • Start on a desktop PC with Ubuntu and the OpenCV samples - Develop your vision algorithms there, first • Use Yocto Open Embedded Linux to build your custom platform • Use the same open source libraries on desktop and target systems - Hacking OpenCV's VideoCapture class will likely be required • Understand which vision components you actually need and why • Select cameras carefully, considering software support - You really need a good V4L2 driver with Yocto recipes - Use a MIPI camera if possible, instead of USB • Explore the acceleration options for your SoC (GPU, IPU, NEON, etc) - Acceleration is required for most CV applications of camera video on ARM 28
  29. 29. © 2019 Montgomery One Resources 29 OpenCV Installation Tutorial https://www.docs.opencv.org/2.4.13/doc/tutorials/intr oduction/linux_install/linux_install.html#linux- installation Yocto Open Embedded Linux https://www.yoctoproject.org/ GStreamer Documentation https://gstreamer.freedesktop.org/documentation/ V4L2 Documentation https://www.kernel.org/doc/html/v4.9/media/uapi/v4l /v4l2.html Khronos OpenGL ES, OpenCL and OpenVX Documentation https://www.khronos.org/ Lightwing Open Mixed-Reality Platform https://montgomery1.com/lightwing/ NXP i.MX6/8 Development Boards https://www.wandboard.org/ OpenCV Status on Github https://github.com/opencv/opencv/wiki/ChangeLog
  30. 30. © 2019 Montgomery One Appendix
  31. 31. © 2019 Montgomery One OpenCV C++ Example - Edge Detection on Camera Video int iEdgeThreshSobel = 1; Mat mImageFrame, mImageGray, mImageGrayBlur, mImageEdgeMask, mImageEdgeResult; const char* WindowName = "Canny edge map with Sobel gradient"; static void onTrackbar(int, void*) { cvtColor(mImageFrame, mImageGray, COLOR_BGR2GRAY); // Create blurred gray scale image for edge detection. blur(mImageGray, mImageGrayBlur, Size(3, 3)); Canny(mImageGrayBlur, mImageEdgeMask, iEdgeThreshSobel, iEdgeThreshSobel * 3, 3); // Canny detector with sobel filter. mImageEdgeResult = Scalar::all(0); // Clear to black. mImageFrame.copyTo(mImageEdgeResult, mImageEdgeMask); imshow(WindowName, mImageEdgeResult); // Display image frame in window. } int main(int argc, char** argv) { VideoCapture capture; capture.open(0); // Open camera device through V4L2. namedWindow(WindowName, WINDOW_KEEPRATIO); // Create window and slider control. createTrackbar("Canny threshold Sobel", WindowName, &iEdgeThreshSobel, 100, onTrackbar); char key = 0; while (key != ‘q’) // Capture frames from the camera and display them. { capture >> mImageFrame; // Capture another image frame from camera. if (mImageFrame.empty()) break; onTrackbar(0, 0); // Show the image. key = (char)waitKey(30); // Wait 30 milliseconds for a key press. } return 0; } 31
  32. 32. © 2019 Montgomery One 32 Lightwing – Open Mixed-Reality Video Wall Platform • Build custom digital signs and video walls with interactive 3D content • Camera motion tracking controls 3D assets without VR headsets or hand controllers • Supports both touch screens and camera motion tracking input • Multipanel video wall dimensions of any size for public spaces • Built-in 3D animations, fonts, effects, audio, video, images and web RSS feeds • GPU and IPU acceleration of camera video on the NXP i.MX6/8 series SoCs • Scriptable, browserless architecture built on open tools for Windows and Linux https://montgomery1.com/lightwing/
  33. 33. © 2019 Montgomery One Lightwing on Linux verses Android and Windows 33
  34. 34. © 2019 Montgomery One About Clay D. Montgomery • 30+ Years of Embedded Multimedia Software Development • C/C++, OpenGL 3D, Audio/Video, Linux, Windows and Android • Previously worked at STB (3Dfx), VLSI (NXP), Nokia, TI and AMX (Harmon) • Authored on Embedded OpenGL ES for Intel Developer Zone, TI, Montgomery One • Freelance Embedded Linux Developer since 2010 • Now working on Open Embedded Linux (Yocto) on ARM almost Exclusively • Active in Tech Start Up Community in Texas • Created Lightwing Mixed-Reality Engine for Interactive Digital Signage, Video Walls and Touch-Kiosks on NXP i.MX6/8 SoCs • Currently adding CV Object Detection and Motion Tracking Features to Lightwing 34

×