Download to read offline
![Faculty of Engineering and Material
Science
Mechatronics Department
The German University in Cairo
Khaled H. Emam 10
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
Converting from u[pixels],v[pixels],Z[mm] coordinates to
X[mm],Y[mm],Z[mm] Cartesian coordinates
Segmentation of objects using depth map](https://image.slidesharecdn.com/15e709f0-d40f-4f53-a644-e3ed060d9494-160216095540/75/Presentation2-10-2048.jpg)
Uploaded byKhaled Emam
Presentation2
This document discusses collision avoidance in CNC machines using multiple sensors. It proposes detecting collisions between the cutting tool and workpiece using a Kinect sensor and two webcams through image processing techniques in MATLAB. The methodology involves extracting depth maps from Kinect to detect the spindle and workpiece, and using SIFT and normalized cross-correlation on webcam images to detect the cutting tool. Adaptive background subtraction will be used to isolate foreground objects. Detected objects across sensors will be fused to determine if a collision has occurred.
Presentation2
- 1. Khaled Hossam Emam Supervisor: Dipl.-Ing. Jens Friedrich
- 2. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 2 CNC Computer Numerical Control Applications and usage Collision avoidance Types of workshop accidents Consequence of accidents and misusage Multiple sensors Cameras Kinect
- 3. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 3 Introduction Kinect sensor Image processing techniques Methodology and code hierarchy Results and conclusion Future work and recommendations Germany and personal experience (2 min)
- 4. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 4 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Motivation Collision avoidance using multiple cameras. Using the theory of sensor fusion and matlab image processing toolbox. Assigned goals Detection of collision between cutting tool and machine table with the workpiece attached Detection of collision between workpiece (clamp) and the machine walls Project deliverables Building a matlab code for collision detection using three cameras.
- 5. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 5 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations
- 6. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 6 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations March April May June July Documentat- ion and literature review Extracting information from Kinect Object detection using Kinect Fuse webcams information with Kinect
- 7. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 7 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Electing a suitable detection algorithm using Kinect for tracking the spindle and machine table including the workpiece. March April May June July Documentat- ion and literature review Extracting information from Kinect Object detection using Kinect
- 8. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 8 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Placement of the two webcams to assist Kinect Detection algorithm for the moving objects using webcams Fuse the webcams information with Kinect sensor March April May June July Fuse webcams information with Kinect
- 9. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 9 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Kinect or Kinect for Xbox 360 is a Controller-free gaming experience launched by Microsoft for Xbox 360. Kinect was one of the used hardware sensors used in the thesis for the interest of generating a depth map of the captured image. Linking the device to matlab and windows Operating system using OpenNI and prime sense drivers Official Microsoft SDK for Kinect Reference :Wikipedia Foundation Inc. (nd.).Retrieved from http://en.wikipedia.org/wiki/Kinect
- 10. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 10 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Converting from u[pixels],v[pixels],Z[mm] coordinates to X[mm],Y[mm],Z[mm] Cartesian coordinates Segmentation of objects using depth map
- 11. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 11 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Digital image Gray level image RGB images Binary image Indexed image Image processing Low level Middle level High level Video processing Object detection Object tracking
- 12. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 12 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations
- 13. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 13 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Proved efficiency over conventional frame differencing , median frame difference and mixture of Gaussians (MOG) background subtraction techniques. No trails are left after a moving object. No noise like moving slightly moving background object Keep information about foreground objects ,when they are not in motion for a convenient time, according to a preset learning rate
- 14. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 14 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Detection stages for SIFT features are: 1. Scale-space extrema detection 2. Keypoint localization 3. Orientation assignment 4. Generation of keypoint descriptors. Matching algorithm using dot product instead of Euclidean distance
- 15. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 15 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Reference: David G. Lowe, "Object recognition from local scale-invariant features," International Conference on Computer Vision, Corfu, Greece (September 1999), pp. 1150-1157.
- 16. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 16 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Consider a template of an object and a frame (image) containing this object, normalized cross correlation is based on padding a template image on the frame. Multiplying both the template and the frame till reaching the maximum value which is the location of the object in the template on the frame. The peak in the figure shows the location of matching between the template and frame. Reference: Mathworks www.mathworks.com
- 17. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 17 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Webcam 2 Webcam 1
- 18. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 18 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Webcam 2 Kinect & Webcam 1
- 19. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 19 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations 1. A saved copy of the black circle on the spindle is used to locate the spindle using normalized cross correlation 2. Using matlab boundary detection function, the boundary of the spindle is located so that lowest part is the cutting tool 3. A copy of the tool is saved to be used in multi-sensor fusion
- 20. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 20 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Webcam detects the cutting tool using SIFT matching between a template of the tool shape captured by Kinect and the frame captured by each webcam. I made of the property of the tool shape that it have similar shape from different viewpoints Webcam 1 - SIFT Webcam 2 - SIFT
- 21. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 21 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Workpiece and machine table are considered to be a single moving object. This object is detected using ABS (Adaptive Background Subtraction) using the Kinect and the two usb-webcams Initial background is already saved for the three cameras with the table is outside the vision of each camera. Thus a background model is initialized without the required object. Background model Current frame Foreground without filtration ‘noise’
- 22. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 22 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Collision between workpiece ‘clamp’ and the machine wall This type of collision is detected using Kinect sensor without the assistance of the webcams. Using the option of camera calibration and converting from pixels coordinates to the Cartesian coordinates with respect to Kinect, location of the workpiece is detected after being detected as a foreground abject with ABS Collision between workpiece and cutting tool A counter is initialized to count for the collision detected by the three cameras. Whenever two cameras detect collision, a warning is given to the operator Collision Algorithm will be discussed with the results and conclusion
- 23. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 23 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Capture rgb and depth frames using Kinect Capture frame using webcam1 Capture frame using webcam2 SIFT matching: webcam1 and tool template SIFT matching: webcam2 and tool template Collision Warning between tool and table ABS to detect table - workpiece Correlation to locate spindle and tool shape Collision warning between workpiece and machine wall Capture rgb and depth frames using Kinect
- 24. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 24 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Detecting this type of collision showed success using Kinect The figure shows the foreground object at collision with the wall Collision is given when the left most point of bounding box of the foreground object is at -500 mm with respect to the Kinect sensor
- 25. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 25 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Workpiece-tool collision is detected using Kinect using the spindle boundary shape During collision the lowest part of the spindle is changed from being the tooltip to be the lowest part of the workpiece. At tool-table collision, the lowest boundary point is the table
- 26. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 26 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations The collision is detected by the interference of the bounding boxes of the spindle and foreground objects At collision, the collision counter is incremented Foreground objects by webcam - error due to occlusion and background change
- 27. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 27 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Full knowledge of the target objects (workpiece and cutting tool) locations and shapes. Data fusion between multi-sensors in order to have a 3D sense of the scene within the CNC workspace in order to detect collision The average processing time of the whole code is 2~3 seconds which is half or one third frames per second due to the limited capabilities of the used computer and matlab is not the optimum program for live image processing .
- 28. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 28 Introduction Kinect sensor Image processing Methodology & matlab code Results and conclusion Future work & recommendations Link between matlab and the CAD/CAM system in order to know the exact coordinates of the moving objects for easier segmentation and more accurate results. Using OpenCV software (Open Source Computer Vision) Exploring Microsoft Kinect SDK, official Kinect driver. More pipelining of the code and using the matlab parallel computation toolbox, in addition to using a faster computer.
- 29. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 29 • Stuttgart, Germany • Student life • Academic atmosphere • Cultural exchange
- 30. Faculty of Engineeringand Material Science Mechatronics Department The German University in Cairo Khaled H. Emam 30 Thanks for listening Your questions or feedback are highly appreciated Khaled Hossam Emam khaled.emam@student.guc.edu.eg