A Three-Dimensional
Representation method for
Noisy Point Clouds based on
Growing Self-Organizing Maps
accelerated on GPUs...
Outline







Introduction
3D Representation using Growing Self-Organizing Maps
Improving keypoint detection from n...
 Introduction
•
•
•
•







Index

Motivation
Framework
Goals

Proposal

3D representation using growing self-organ...
Introduction

Motivation

Motivation


Most computer vision problems require the use of an
effective way of representatio...
Introduction

Motivation

Motivation (II)







3D data captured from the real world

•

Implicitly is comprised of c...
Introduction



Motivation

Motivation (III)
3D models of objects and scenes have been extensively used in
computer graph...
Introduction

Framework

Framework






Regional research project (GV/2011/034)

•

Title: “Visual surveillance syste...
Introduction

Goals

Goals

 Proposal and validation of a 3D representation
model and a data fitting algorithm for noisy
...
Introduction

Goals

Goals (II)

 Validation of the proposed method on different
real computer vision problems handling 3...
Introduction

Proposal

Proposal

 Growing Self-Organizing Maps (GSOM) for 3D
data representation

•
•
•

Low cost 3D sen...
Index


Introduction



3D Representation using Growing Self-Organizing Maps






•
•
•
•

Review
3D Growing Neural...
3D Representation GSOM

Review

Review


SOMs were originally proposed for data clustering and pattern
recognition purpos...
3D Representation GSOM

Review

Review (II)


Moreover, there exist some limitations and unexplored
topics in the applica...
3D Representation GSOM

GNG network

3D Growing Neural Gas Network






Obtaining a reduced and compact representation...
3D Representation GSOM

GNG algorithm

GNG algorithm


Input data is defined in ℝ 𝒅

•



Adaption: Reconfiguration modu...
3D Representation GSOM

Experiments: Data

Experiments: Data acquisition



Algorithm independent of the data source
We ...
3D Representation GSOM

Experiments: Data

Experiments: Data acquisition

3D sensors used for experiments. From left to ri...
3D Representation GSOM

Experiments: Data

Experiments: Data Sets

 Some public data sets have been used to validate
the ...
3D Representation GSOM

Experiments: Data

Experiments: Data Sets


Blensor software: It allowed us to generate synthetic...
3D Representation GSOM

Experiments

Experiments

 GNG method has been applied to 3D data
representation

•
•

Input spac...
3D Representation GSOM

Exp: GNG 3D representation

Experiments: GNG 3D representation

Applying GNG to laser data

Applyi...
3D Representation GSOM

Exp: GNG 3D representation

Experiments: GNG 3D representation

Applying GNG to Kinect data
22/79
3D Representation GSOM

Exp: Input space Adaptation

Experiments: Input space adaptation



GNG method obtains better ada...
3D Representation GSOM

Exp: Input space Adaptation

Experiments: Input space adaptation (II)

Noisy model σ = 0.4

GNG re...
3D Representation GSOM

Exp: Normal estimation

Experiments: Normal estimation




Surface normals are important propert...
3D Representation GSOM

Exp: Input space Adaptation

Experiments: Normal estimation (II)

Top:. Normal estimation on a fil...
3D Representation GSOM

Extensions: Colour-GNG

Extension: Colour-GNG



Modern 3D sensors provide colour information (e...
3D Representation GSOM

Extensions: Colour-GNG

Extension: Colour-GNG (II)

(a),(b),(c) show original point clouds. (d),(e...
3D Representation GSOM

Extensions: Colour-GNG

Extension: Colour-GNG (III)




Mario figure is down-sampled using the C...
3D Representation GSOM

Extensions: Sequences

Extension: Sequences management





Extension of the GNG for
processin...
3D Representation GSOM

Extension: 3D Reconstruction

Extension: 3D Surface Reconstruction



Three-dimensional surface ...
3D Representation GSOM

Extension: 3D Reconstruction

Extension: 3D Surface Reconstruction (II)


Avoid non-manifold and
...
3D Representation GSOM

Extension: 3D Reconstruction

Extension: 3D Surface Reconstruction (III)
Left: The triangle formed...
3D Representation GSOM

Extension: 3D Reconstruction

Extension: 3D Surface Reconstruction (IV)

Different views of recons...
3D Representation GSOM

Extension: 3D Reconstruction

Extension: 3D Surface Reconstruction (V)

Reconstructed models using...
3D Representation GSOM

Extension: 3D Reconstruction

Extension: 3D Surface Reconstruction (VI)
Top: 3D model of a person
...
Index








Introduction
3D Representation using Growing Self-Organizing Maps

Improving keypoint detection from n...
Improving Keypoint detection



Review

Review
Filtering and down-sampling have become essential steps in
3D data process...
Improving Keypoint detection

Review

Review (II)
Registration: Aligning various 3D point cloud data views into a complete...
Improving Keypoint detection

Keypoint detection

3D Keypoint detection
 Applying keypoint detection algorithms to filter...
Improving Keypoint detection

Keypoint detection

3D Keypoint detection (II)
 3D Keypoint detectors

• SIFT3D:

using dep...
Improving Keypoint detection

Feature descriptors

3D Feature descriptors
 Feature descriptors are calculated over detect...
Improving Keypoint detection

Feature matching

Feature matching (II)
 Correspondences between keypoints are validated
th...
Improving Keypoint detection

Results

Results: Feature matching






Correspondences
matching computed on
different...
Improving Keypoint detection

Results

Results: Transformation errors
Lowest max errors
Lowest transformation error

*1

M...
Index









Introduction
3D Representation using Growing Self-Organizing Maps
Improving keypoint detection from n...
GPGPU Implementations

GPUs

Graphics Processing Unit
 GPUs have democratized High Performance
Computing (HPC)
• Massivel...
GPGPU Implementations

GPUs

Graphics Processing Unit (II)






More transistors for data processing
GPU are compris...
GPGPU Implementations

GPU Implementation GNG

GPU Implementation GNG
 Stages of the GNG algorithm
that are highly parall...
GPGPU Implementations

GPU Implementation GNG

Parallel Min/Max Reduction







A parallel Min/Max reduction
that co...
GPGPU Implementations

Experimental Setup

Experimental setup

 Main GNG Parameters:

•
•

~0-20,000 neurons and a maximu...
GPGPU Implementations

Exp: 2MinParallelReduction

Experiments: 2MinParallelReduction

Runtime and speed-up using CPU and ...
GPGPU Implementations

Exp: GNG Runtime

Experiments: GNG learning runtime

GPU and CPU GNG runtime, and speed-up for diff...
GPGPU Implementations

Exp: Hybrid version

Experiments: Hybrid version


CPU implementation was faster for small network...
GPGPU Implementations

GPU Feature extraction

GPU-based Tensor extraction algorithm
 Time-constrained 3D feature extract...
GPGPU Implementations

GPU Feature extraction

GPU-based Tensor extraction algorithm (II)





The surface area of the ...
GPGPU Implementations

Exp: performance

Experiments: Performance

Runtime comparison and speed-up obtained for proposed m...
Index









Introduction
3D Representation using Growing Self-Organizing Maps
Improving keypoint detection from n...
Applications

Exp: performance

Applications

 Different cases of study where the GNG-based method
proposed in this PhD t...
Applications

Robotics

Robotics: 6DoF pose registration
 The main goal of this application is to perform six degrees
of ...
Applications

Robotics

Robotics: 6DoF pose registration (II)

Without GNG




GNG

Left: planar patches extracted from ...
Applications

Robotics

Robotics: 6DoF pose registration (III)

Robot trajectory

Without GNG




GNG

Planar based 6DoF...
Applications

3D Object recognition

3D Object Recognition
 The main goal of this
application is the recognition
of objec...
Applications

3D Object recognition

3D Object Recognition (II)

 Object recognition is
performed on scenes
with differen...
Applications

3D Object recognition

3D Object Recognition (III)




GPU-based 3D feature implementation is successfull...
Applications

CAD: Rapid Prototyping

Rapid Prototyping in Shoe Last Manufacturing

 With

the advent of CAD/CAM and rapi...
Applications

CAD: Rapid Prototyping

Rapid Prototyping in Shoe Last Manufacturing (II)



The main goal of this research...
Applications

CAD: Rapid Prototyping

Rapid Prototyping in Shoe Last Manufacturing (III)

Voxel Grid versus GNG: Mean erro...
Applications

CAD: Rapid Prototyping

Rapid Prototyping in Shoe Last Manufacturing (IV)

Input space
GNG nodes
VG nodes

G...
Index








Introduction
3D Representation using Growing Self-Organizing Maps
Improving keypoint detection from no...
Conclusions

Contributions

Contributions

 Contributions made in the topic of research:
•

Proposal of a new method to c...
Conclusions

•

Contributions

Contributions (II)
Integration of 3D data processing algorithms in complex
computer vision ...
Conclusions

Future work

Future work

 Other improvements on the GPU implementation of the
GNG algorithm:

•

•
•
•

Usi...
Conclusions

Publications

Publications

•

4 JCR Journal papers
o

“Real-time 3D semi-local surface patch extraction usin...
Conclusions

Publications

Publications (II)

•

International conferences
o

“Point Light Source Estimation based on Scen...
Conclusions

Publications

Publications (III)

•

International conferences
o

“Improving 3D Keypoint Detection from Noisy...
Conclusions

Publications

Publications (IV)

•

International conferences
o

“A study of registration techniques for 6DoF...
Conclusions

Publications

Publications (V)

•

National conferences
o

“Procesamiento de múltiples flujos de datos con Gr...
This presentation is licensed under a Creative Commons AttributionNonCommercial-ShareAlike 4.0 International License.

79/...
A Three-Dimensional
Representation method for
Noisy Point Clouds based on
Growing Self-Organizing Maps
accelerated on GPUs...
Upcoming SlideShare
Loading in …5
×

A Three-Dimensional Representation method for Noisy Point Clouds based on Growing Self-Organizing Maps accelerated on GPUs

1,084 views
971 views

Published on

Slides used for the thesis defense of the PhD candidate Sergio Orts-Escolano.

The research described in this thesis was motivated by the need of a robust model capable of representing 3D data obtained with 3D sensors, which are inherently noisy. In addition, time constraints have to be considered as these sensors are capable of providing a 3D data stream in real time.This thesis proposed the use of Self-Organizing Maps (SOMs) as a 3D representation model. In particular, we proposed the use of the Growing Neural Gas (GNG) network, which has been successfully used for clustering, pattern recognition and topology representation of multi-dimensional data. Until now, Self-Organizing Maps have been primarily computed offline and their application in 3D data has mainly focused on free noise models, without considering time constraints. It is proposed a hardware implementation leveraging the computing power of modern GPUs, which takes advantage of a new paradigm coined as General-Purpose Computing on Graphics Processing Units (GPGPU). The proposed methods were applied to different problems and applications in the area of computer vision such as the recognition and localization of objects, visual surveillance or 3D reconstruction.

Published in: Technology, Art & Photos
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,084
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

A Three-Dimensional Representation method for Noisy Point Clouds based on Growing Self-Organizing Maps accelerated on GPUs

  1. 1. A Three-Dimensional Representation method for Noisy Point Clouds based on Growing Self-Organizing Maps accelerated on GPUs Author: Sergio Orts Escolano Supervisors: Dr. José García Rodríguez Dr. Miguel Ángel Cazorla Quevedo Doctoral programme in technologies for the information society
  2. 2. Outline       Introduction 3D Representation using Growing Self-Organizing Maps Improving keypoint detection from noisy 3D observations GPGPU parallel implementations Applications Conclusions 2/79
  3. 3.  Introduction • • • •      Index Motivation Framework Goals Proposal 3D representation using growing self-organizing maps Improving keypoint detection from noisy 3D observations GPGPU parallel Implementations Applications Conclusions 3/79
  4. 4. Introduction Motivation Motivation  Most computer vision problems require the use of an effective way of representation • • Graphs, regions of interest (ROI), B-Splines, Octrees, histograms, … Key step for later processing stages: feature extraction, feature matching, classification, keypoint detection, … 4/79
  5. 5. Introduction Motivation Motivation (II)     3D data captured from the real world • Implicitly is comprised of complex structures and nonlinear models The advent of low cost 3D sensors • • • E.g. Microsoft Kinect, Asus Xtion, PrimeSense Carmine, … RGB and Depth (RGB-D) streams (25 frames per second (fps)) High levels of noise and outliers Only few works in 3D computer vision deal with real-time constraints • 3D data processing algorithms are computationally expensive Finding a 3D model with different features: • • • • Rapid adaptation Good quality of representation (topology preservation) Flexibility (non-stationary data) Robust to noisy data and outliers 5/79
  6. 6. Introduction  Motivation Motivation (III) 3D models of objects and scenes have been extensively used in computer graphics • • • Suitable structure for rendering and display Common graphics representations include quadric surfaces [Gotardo et al., 2004], B-spline surfaces [Gregorski et al., 2000], and subdivision surfaces Not general enough to handle such a variety of features: flexibility, adaption, noise-aware, … that are present in computer vision problems (Left) Point cloud captured from a manufactured object (builder helmet). (Right) 3D mesh generated from the captured point cloud (post-processed) 6/79
  7. 7. Introduction Framework Framework     Regional research project (GV/2011/034) • Title: “Visual surveillance systems for the identification and characterization of anomalous behaviour” Project financed by the Valencian Government in Spain Regional research project (GRE09-16) • Title: “Visual surveillance systems for the identification and characterization of anomalous behaviour in restricted environments under temporal constraints” Project financed by the University of Alicante in Spain National research project (DPI2009-07144) • Title: “Cooperative Simultaneous Localization and Mapping (SLAM) in large scale environments” Research stay at IPAB – University of Edinburgh (BEFPI/2012/056) • Title: “Real-time 3D feature estimation and keypoint detection of scenes using GPGPUs” 7/79
  8. 8. Introduction Goals Goals  Proposal and validation of a 3D representation model and a data fitting algorithm for noisy point clouds • • • • Deals with noisy data and outliers Flexible Dynamic (non stationary data) Topology preserving  An accelerated hardware implementation of the proposed technique • • Considerable speed-up regard CPU implementations Real-time frame rates 8/79
  9. 9. Introduction Goals Goals (II)  Validation of the proposed method on different real computer vision problems handling 3D data: • • • Robot vision: 6DoF Egomotion 3D Object recognition Computed-aided design/manufacturing (CAD/CAM)  Integration of 3D data processing algorithms in complex computer vision systems • • Filtering, downsampling, normal estimation, feature extraction, keypoint detection, matching, … The use of a GPU as a general purpose processor 9/79
  10. 10. Introduction Proposal Proposal  Growing Self-Organizing Maps (GSOM) for 3D data representation • • • Low cost 3D sensors: noisy data Time-constrained conditions Applications: 3D computer vision problems  Hardware-based implementation of the proposed GSOM method • General Purpose computing on Graphics Processing Units (GPGPU) paradigm  Integrate the entire pipeline of 3D computer vision systems using GPGPU paradigm 10/79
  11. 11. Index  Introduction  3D Representation using Growing Self-Organizing Maps     • • • • Review 3D Growing Neural Gas network Experiments: Input space adaptation & normal estimation Extensions of the GNG algorithm Improving keypoint detection from noisy 3D observations GPGPU Parallel Implementations Applications Conclusions 11/79
  12. 12. 3D Representation GSOM Review Review  SOMs were originally proposed for data clustering and pattern recognition purposes [Kohonen, 1982, Vesanto and Alhoniemi, 2000, Dittenbach et al., 2001]   As the original model had some drawbacks due to the preestablished topology of the network, growing approaches were proposed in order to deal with this problem Growing Neural Gas network has been successfully applied to the representation of 2D shapes in many computer vision problems [Stergiopoulou and Papamarkos, 2006, García-Rodríguez et al., 2010, Baena et al., 2013]  Already exist approaches that use traditional SOMs for 3D data representation: [Yu, 1999, Junior et al., 2004] • • • Difficulties to correctly approximate concave structures High computational cost Synthetic data 12/79
  13. 13. 3D Representation GSOM Review Review (II)  Moreover, there exist some limitations and unexplored topics in the application of SOM-based methods to 3D representation: • • • • • Majority of these works do not consider the high computational cost of the learning step Do not guarantee response within strict time constraints Assumed perfect point clouds that were noise-free Data fusion (Geometric information + colour information) has not been considered Not dealing with point cloud sequences, only singleshot data 13/79
  14. 14. 3D Representation GSOM GNG network 3D Growing Neural Gas Network    Obtaining a reduced and compact representation of 3D data • Self Organizing Maps – Growing Neural Gas Growing Neural Gas Algorithm (GNG) [Fritzke, 1995] • • • • Incremental training algorithm Links between the units in the network are established through Competitive Hebbian Learning (CHL) Topology Preserving Graph Flexibility, growth, rapid adaption and good quality of representation. GNG representation is comprised of nodes (neurons) and connections (edges) • Wire-frame model Initial, intermediate and final states of the GNG learning algorithm 14/79
  15. 15. 3D Representation GSOM GNG algorithm GNG algorithm  Input data is defined in ℝ 𝒅 •  Adaption: Reconfiguration module •  For 3D representation d=3 Random patterns are presented to the network Growth: It starts with two neurons and new neurons are inserted  Flexibility: neurons and connections may be removed during the learning stage  This process is repeated until an ending condition is fulfilled: • • Number of neurons/patterns Adaptation error threshold Highly Parallelizable 15/79
  16. 16. 3D Representation GSOM Experiments: Data Experiments: Data acquisition   Algorithm independent of the data source We managed 3D data that can come from different sensors • Laser unit, a LMS-200 Sick mounted on a sweeping unit. o Outdoor environments. Its range is 80 metres with an error of 1 millimetre per metre • Time-of-Flight camera o SR4000 camera. It has a range of 5-10 metres o The accuracy varies depending on the characteristics of the observed scene, such as objects reflectivity and ambient lighting conditions o Generation of point clouds during real time acquisition • Range camera: structured light, Microsot Kinect device o RGB-D information. Indoor environments. Its range is from 0.8 to 6 metres o Generation of point clouds during real time acquisition 16/79
  17. 17. 3D Representation GSOM Experiments: Data Experiments: Data acquisition 3D sensors used for experiments. From left to right: Sick laser unit LMS-200, Time-Of-Flight SR4000 camera and Microsoft Kinect Mobile robots used for experiments. Left: Magellan Pro unit used for indoors. Right: PowerBot used for outdoors. 17/79
  18. 18. 3D Representation GSOM Experiments: Data Experiments: Data Sets  Some public data sets have been used to validate the proposed method: • • • Well known Stanford 3D scanning repository. It contains complete models that have been previously processed (noise free) Dataset captured using the Kinect sensor. Released by the Computer Vision Laboratory of the University of Bologna [Tombari et al., 2010a] Own dataset obtained using three previously mentioned 3D sensors 18/79
  19. 19. 3D Representation GSOM Experiments: Data Experiments: Data Sets  Blensor software: It allowed us to generate synthetic scenes and to obtain partial views of the generated scene as if a Kinect device was used • It provided us with ground truth information for experiments Simulated scene Simulated scene + Gaussian noise 19/79
  20. 20. 3D Representation GSOM Experiments Experiments  GNG method has been applied to 3D data representation • • Input space adaptation Noise removal properties  Extensions of the GNG based algorithm • • • Colour-GNG Sequences management 3D Surface reconstruction 20/79
  21. 21. 3D Representation GSOM Exp: GNG 3D representation Experiments: GNG 3D representation Applying GNG to laser data Applying GNG to Kinect data Applying GNG to SR4000 data 21/79
  22. 22. 3D Representation GSOM Exp: GNG 3D representation Experiments: GNG 3D representation Applying GNG to Kinect data 22/79
  23. 23. 3D Representation GSOM Exp: Input space Adaptation Experiments: Input space adaptation  GNG method obtains better adaptation to the input space than other filtering methods like Voxel Grid technique • Obtains lower adaptation (Mean Squared Error (MSE) ) • Tested on CAD models and simulated scenes Lower error Input space adaptation MSE for different models (metres). Voxel Grid versus GNG. Numbers in bold provide the best results. 23/79
  24. 24. 3D Representation GSOM Exp: Input space Adaptation Experiments: Input space adaptation (II) Noisy model σ = 0.4 GNG representation Original CAD model Voxel grid representation Filtering quality using 10,000 nodes. GNG vs Voxel Grid comparison 24/79
  25. 25. 3D Representation GSOM Exp: Normal estimation Experiments: Normal estimation   Surface normals are important properties of a geometric surface, and are heavily used in many areas such as computer vision and computer graphics Normal or curvature estimation can be affected by the presence of noise Normal estimation over noisy input data  Representation obtained using the GNG method allows to compute more accurate normal information 25/79
  26. 26. 3D Representation GSOM Exp: Input space Adaptation Experiments: Normal estimation (II) Top:. Normal estimation on a filtered point cloud produced by the GNG method. Bottom: Normal estimation on a raw point cloud.  Normals are considered more stable as their distribution is smooth and also they have less abrupt changes in their directions 26/79
  27. 27. 3D Representation GSOM Extensions: Colour-GNG Extension: Colour-GNG   Modern 3D sensors provide colour information (e.g. Kinect, Carmine, Asus Xtion, … ) GNG is extended considering colour information during the learning step • • • • Input data is defined in ℝ 𝒅 where d = 6 Colour information is considered during the weight adaptation step but it was not included in the CHL (winning neurons) process o We are still focus on topology preservation Winning neuron step only compute Euclidean distance using x,y,z components No post-processing steps are required as neurons’ colour is obtained during the learning process 27/79
  28. 28. 3D Representation GSOM Extensions: Colour-GNG Extension: Colour-GNG (II) (a),(b),(c) show original point clouds. (d),(e),(f) show downsampled point clouds using the proposed method 28/79
  29. 29. 3D Representation GSOM Extensions: Colour-GNG Extension: Colour-GNG (III)   Mario figure is down-sampled using the Colour-GNG method Results are similar to those obtained with the colour interpolation post-processing step 29/79
  30. 30. 3D Representation GSOM Extensions: Sequences Extension: Sequences management     Extension of the GNG for processing sequences of point clouds It is not required to restart the learning It provides a speed-up in the runtime as neurons are kept between point clouds This extension was applied in a mobile robotics application An improved workflow to manage point cloud sequences using the GNG algorithm 30/79
  31. 31. 3D Representation GSOM Extension: 3D Reconstruction Extension: 3D Surface Reconstruction   Three-dimensional surface reconstruction is not considered in the original GNG algorithm as it only generates wire-frame models [Holdstein and Fischer, 2008, Do Rego et al., 2010, Barhak, 2002] have already considered the creation of 3D triangular faces modifying the original GNG algorithm •  Post-processing steps are required for avoid gaps and holes in the final mesh We extended the CHL developing a method able to produce full 3D meshes during the learning stage • • No post-processing steps are required A new learning scheme was developed 31/79
  32. 32. 3D Representation GSOM Extension: 3D Reconstruction Extension: 3D Surface Reconstruction (II)  Avoid non-manifold and overlapping edges • •   More than 2 neighbours it is checked if the face to be created already exist A face is created whenever the already existing edges or the new ones form a triangle The neuron insertion process was also modified Considered situations for edge and face creation during the extended CHL 32/79
  33. 33. 3D Representation GSOM Extension: 3D Reconstruction Extension: 3D Surface Reconstruction (III) Left: The triangle formed by these 3 neurons is close to a right triangle Right: The edge connecting s1 and ni is removed as the angle formed Edge removal constraint based on the Tales sphere Left: neuron insertion between the neuron q with highest error and its neighbour f with highest error. Right: four new triangles and two edges are created considering r, q and f. Face creation during the insertion of new neurons 33/79
  34. 34. 3D Representation GSOM Extension: 3D Reconstruction Extension: 3D Surface Reconstruction (IV) Different views of reconstructed models using an existing GNGbased method [Do Rego et al., 2010] for surface reconstruction  Post-processing steps were avoided causing gaps and holes in the final 3D reconstructed models 34/79
  35. 35. 3D Representation GSOM Extension: 3D Reconstruction Extension: 3D Surface Reconstruction (V) Reconstructed models using our extended GNG method for face reconstruction and without applying post-processing steps 35/79
  36. 36. 3D Representation GSOM Extension: 3D Reconstruction Extension: 3D Surface Reconstruction (VI) Top: 3D model of a person (Kinect sensor). Bottom: digitized foot. (foot digitizer) Left: Noisy point clouds captured using the Kinect sensor. Right: 3D reconstruction using the proposed method. 36/79
  37. 37. Index       Introduction 3D Representation using Growing Self-Organizing Maps Improving keypoint detection from noisy 3D observations • • • • Review Improving keypoint detection Correspondences matching Results GPGPU Parallel Implementations Applications Conclusions 37/79
  38. 38. Improving Keypoint detection  Review Review Filtering and down-sampling have become essential steps in 3D data processing General System Overview Motivation: dealing with noisy data obtained from 3D sensors as the Microsoft Kinect or lasers  Result: Improving 3D keypoint detection and therefore registration problem We propose the use of the GNG algorithm for downsampling and filtering 3D data  Beneficial attributes will be demonstrated through the 3D registration problem 38/79
  39. 39. Improving Keypoint detection Review Review (II) Registration: Aligning various 3D point cloud data views into a complete model Pairwise matching 39/79
  40. 40. Improving Keypoint detection Keypoint detection 3D Keypoint detection  Applying keypoint detection algorithms to filtered point clouds  State-of-the-art 3D keypoint detectors • Different techniques are used to test and measure the improvement achieved using GNG method to filter and downsample input data 40/79
  41. 41. Improving Keypoint detection Keypoint detection 3D Keypoint detection (II)  3D Keypoint detectors • SIFT3D: using depth as the intensity value in the original SIFT algorithm • Harris3D: use surface normals of 3D points • Tomasi3D: performs eigenvalue decomposition over covariance matrix • Noble3D: evalutes the ratio between the determinant and the trace of the covariance matrix 41/79
  42. 42. Improving Keypoint detection Feature descriptors 3D Feature descriptors  Feature descriptors are calculated over detected keypoints to perform feature matching • • FPH and FPFH: based on an histogram of the differences of angles between the normals of the neighbour points SHOT and CSHOT: a spherical grid centered on the point divides the neighbourhood so that in each grid bin a weighted histogram of normals is obtained FPFH CSHOT 42/79
  43. 43. Improving Keypoint detection Feature matching Feature matching (II)  Correspondences between keypoints are validated through RANSAC algorithm, rejecting those inconsistent correspondences 43/79
  44. 44. Improving Keypoint detection Results Results: Feature matching      Correspondences matching computed on different input data Top: raw point clouds Middle: reduced representation using the GNG (20, 000 neurons) Bottom: reduced representation using the GNG (10, 000 neurons) RANSAC is used to reject wrong matches Raw 3D data GNG 20,000 nodes GNG 10,000 nodes 44/79
  45. 45. Improving Keypoint detection Results Results: Transformation errors Lowest max errors Lowest transformation error *1 Mean, median, minimum and maximum RMS*2 errors of the estimated transformations using different keypoint detectors. (metres). *1 Uniform Sampling *2 Root Mean Square - Transformation error 45/79
  46. 46. Index       Introduction 3D Representation using Growing Self-Organizing Maps Improving keypoint detection from noisy 3D observations GPGPU Parallel Implementations • • • • Graphics Processing Unit GPU-based implementation of the GNG algorithm GPU-based tensor extraction algorithm Conclusions Applications Conclusions 46/79
  47. 47. GPGPU Implementations GPUs Graphics Processing Unit  GPUs have democratized High Performance Computing (HPC) • Massively parallel processors on a commodity PC • Great ratio FLOP/€ compared with other solutions  However, this is not for free • New programming model • Algorithms need to be re-thought and re-implemented  Growing Neural Gas algorithm is computationally expensive • Most computer vision applications are time-constrained • A GPGPU implementation is proposed 47/79
  48. 48. GPGPU Implementations GPUs Graphics Processing Unit (II)      More transistors for data processing GPU are comprised of streaming multi-processors High GPU Memory bandwidth GPGPU: General Purpose computing on Graphics Processors Units Key hardware feature is that the cores are SIMT • Single instruction multiple threads G80 CUDA NVIDIAs Architecture 48/79
  49. 49. GPGPU Implementations GPU Implementation GNG GPU Implementation GNG  Stages of the GNG algorithm that are highly parallelizable • • • • Calculate distance to neurons for every pattern Search winning neurons Delete neurons and edges Search neuron with max error  Other improvements • • Avoid memory transfers between CPU and GPU Hierarchy of memories Highly parallelizable stages 49/79
  50. 50. GPGPU Implementations GPU Implementation GNG Parallel Min/Max Reduction      A parallel Min/Max reduction that computes the Min/Max of large arrays of values (Neurons) Strategy used to find Min/Max winning neurons Reduce linear computational cost n of the sequential version to the logarithmic cost log(n) Provides better performance for a large number of neurons Example of Parallel Reduction Algorithm Proposed version: 2MinParallelReduction • Extended version to obtain 2 minimum values in the same number of steps 50/79
  51. 51. GPGPU Implementations Experimental Setup Experimental setup  Main GNG Parameters: • • ~0-20,000 neurons and a maximum λ (entries per iteration) of 1,000-2,000 Others parameters have been fixed based on previous works [García-Rodriguez et al., 2012] o єw = 0.1 , єn = 0.001 o amax = 250, α = 0.5 , β = 0.0005  Hardware • GPUs: CUDA capable devices used in experiments • CPU: single thread and multiple thread implementations were tested o Intel Core i3 540 3.07Ghz 51/79
  52. 52. GPGPU Implementations Exp: 2MinParallelReduction Experiments: 2MinParallelReduction Runtime and speed-up using CPU and GPU implementations 52/79
  53. 53. GPGPU Implementations Exp: GNG Runtime Experiments: GNG learning runtime GPU and CPU GNG runtime, and speed-up for different devices 53/79
  54. 54. GPGPU Implementations Exp: Hybrid version Experiments: Hybrid version  CPU implementation was faster for small network sizes • • We developed an hybrid implementation GPU version automatically starts computing when it is detected that computing time is lower than the one obtained by the CPU Example of CPU and Hybrid GNG runtime for different devices 54/79
  55. 55. GPGPU Implementations GPU Feature extraction GPU-based Tensor extraction algorithm  Time-constrained 3D feature extraction • Most feature descriptors cannot be computed online due to their high computational complexity o 3D Tensor - [Mian et al., 2006b] o Geometric Histogram - [Hetzel et al., 2001] • • • Highly parallelizable Geometrical properties Invariant to linear transformations o Spin Images - [Andrew Johnson, 1997]  An accelerated GPU-based implementation of an existing 3D feature extraction algorithm is proposed • Accelerate entire pipeline of RGB-D based computer vision systems 55/79
  56. 56. GPGPU Implementations GPU Feature extraction GPU-based Tensor extraction algorithm (II)    The surface area of the mesh intersecting each bin of the grid is the value of the tensor element As many threads as voxels are launched in parallel where each GPU thread represent a voxel (bin) of the grid Each thread computes the area of intersection between the mesh and its corresponding voxel using Sutherland Hodgman’s polygon clipping algorithm. [Foley et al., 1990] 56/79
  57. 57. GPGPU Implementations Exp: performance Experiments: Performance Runtime comparison and speed-up obtained for proposed methods using different graphics boards 57/79
  58. 58. Index       Introduction 3D Representation using Growing Self-Organizing Maps Improving keypoint detection from noisy 3D observations GPGPU Parallel Implementations Applications • • • Robotics Computer Vision CAD/CAM Conclusions 58/79
  59. 59. Applications Exp: performance Applications  Different cases of study where the GNG-based method proposed in this PhD thesis was applied to different areas • Robot Vision o • Computer Vision o • 6DoF Pose Registration 3D object recognition under cluttered conditions CAD/CAM o Rapid Prototyping in Shoe Last Manufacturing 59/79
  60. 60. Applications Robotics Robotics: 6DoF pose registration  The main goal of this application is to perform six degrees of freedom (6DoF) pose registration in semi-structured environments •  We combined our accelerated GNG-based algorithm with the method proposed in [Viejo and Cazorla, 2013] •  Man-made indoor and outdoor environments Planar patches extraction It provides a good starting point for Simultaneous Location and Mapping (SLAM)  GNG was applied directly to raw 3D data 60/79
  61. 61. Applications Robotics Robotics: 6DoF pose registration (II) Without GNG   GNG Left: planar patches extracted from SR4000 camera Right: filtered data using the GNG network: more planar patches are extracted 61/79
  62. 62. Applications Robotics Robotics: 6DoF pose registration (III) Robot trajectory Without GNG   GNG Planar based 6DoF pose registration results Left image shows map building results without using GNG while the results shown on the Right are obtained after computing a GNG mesh 62/79
  63. 63. Applications 3D Object recognition 3D Object Recognition  The main goal of this application is the recognition of objects under time constraints and cluttered conditions  The GPU-based of the semilocal surface feature (tensor) is successfully used to recognize objects in cluttered scenes  A library of models is constructed offline, storing all extracted 3D tensors in an efficient way using a hash table • Multiple views 63/79
  64. 64. Applications 3D Object recognition 3D Object Recognition (II)  Object recognition is performed on scenes with different level of occlusion  Objects are occluded by other objects stored and non-stored in the library  The averaged recognition rate was 84%, wrong matches 16% and false negatives 0% 64/79
  65. 65. Applications 3D Object recognition 3D Object Recognition (III)    GPU-based 3D feature implementation is successfully used in a 3D object recognition application Parallel matching is performed on the GPU: correlation function Implemented prototype took around 800 ms with a GPU implementation to perform 3D object recognition of the entire scene Scene 2 65/79
  66. 66. Applications CAD: Rapid Prototyping Rapid Prototyping in Shoe Last Manufacturing  With the advent of CAD/CAM and rapid acquisition devices it is possible to digitize old raised shoe lasts for reusing them in the shoe last design software Process to reconstruct existing shoe lasts and computing topology preservation error regard the original CAD design 66/79
  67. 67. Applications CAD: Rapid Prototyping Rapid Prototyping in Shoe Last Manufacturing (II)  The main goal of this research is to obtain a grid of points that is adapted to the topology of the footwear shoe last from a sequence of sections with disorganized points acquired by sweeping an optical laser digitizer Typical sequence of sections of a shoe last. Noisy data obtained from the digitizer 67/79
  68. 68. Applications CAD: Rapid Prototyping Rapid Prototyping in Shoe Last Manufacturing (III) Voxel Grid versus GNG: Mean error along different sections of the shoe last 68/79
  69. 69. Applications CAD: Rapid Prototyping Rapid Prototyping in Shoe Last Manufacturing (IV) Input space GNG nodes VG nodes GNG vs VG topological preservation comparison 3D Reconstruction GNG 3D Reconstruction VG 69/79
  70. 70. Index       Introduction 3D Representation using Growing Self-Organizing Maps Improving keypoint detection from noisy 3D observations GPGPU Parallel Implementations Applications Conclusions • • • Contributions Future work Publications 70/79
  71. 71. Conclusions Contributions Contributions  Contributions made in the topic of research: • Proposal of a new method to create compact, reduced and efficient 3D representations from noisy data ‐ Development of a GNG-based method capable to deal with different sensors ‐ ‐ ‐ ‐ Extension of the GNG algorithm to consider colour information ‐ GPU-based implementation to accelerate the learning process of the GNG and NG algorithms. ‐ An hybrid implementation of the GNG algorithm that takes advantage of the CPU and GPU processors Extension of the GNG algorithm for 3D surface reconstruction Sequences management Integration of the proposed method in 3D keypoint detection algorithms improving their performance 71/79
  72. 72. Conclusions • Contributions Contributions (II) Integration of 3D data processing algorithms in complex computer vision systems: ‐ ‐ Point cloud triangulation has been ported to the GPU accelerating its runtime ‐ • Normal estimation has been ported to the GPU considerably decreasing its runtime A GPU time-constrained implementation of a 3D feature extraction algorithm Application of the proposed method in various real computer vision applications: ‐ ‐ Robotics: Localization and mapping: 6DoF pose registration Computer vision: 3D object recognition under cluttered conditionsCAD/CAM: rapid prototyping in shoe last manufacturing 72/79
  73. 73. Conclusions Future work Future work  Other improvements on the GPU implementation of the GNG algorithm: • • • • Using multi-GPU to manage several neural networks simultaneously Distributed computing Testing new architectures: Intel Xeon Phi [Fang et al., 2013a] Generating random patterns using GPU  More applications of the accelerated GNG algorithm will be studied in the future • • Clustering multi-dimensional data: Big Data Medical Image Reconstruction  Extension of the real-time implementation of the 3D tensor • • Visual features extracted from RGB information Improve implicit keypoint detector used by the 3D tensor 73/79
  74. 74. Conclusions Publications Publications • 4 JCR Journal papers o “Real-time 3D semi-local surface patch extraction using GPGPU” S. Orts-Escolano, V. Morell, J. Garcia-Rodriguez, M. Cazorla, R.B. Fisher; Journal of Real-Time Image Processing. December 2013; ISSN: 1861-8219; Impact Factor: 1.156 (JCR 2012) o “GPGPU implementation of growing neural gas: Application to 3D scene reconstruction” S. Orts, J. García Rodríguez, D. Viejo, M. Cazorla, V. Morell; J.Parallel Distrib. Comput. 72(10); pp: 1361-1372 (2012); ISSN: 0743-7315; ImpactFactor: 1.135 (JCR 2011) o “3D-based reconstruction using growing neural gas landmark: application to rapid prototyping in shoe last manufacturing” A. Jimeno-Morenilla, J. García-Rodriguez, S. Orts-Escolano, M. Davia-Aracil; The International Journal of Advanced Manufacturing Technology: May 2013. Vol 69. pp: 657-668; ISSN: 0268-3768; Impact Factor: 1.205 (JCR 2012) o “Autonomous Growing Neural Gas for applications with time constraint: Optimal parameter estimation” J. García Rodríguez, A. Angelopoulou, J. M. García Chamizo, A. Psarrou, S. OrtsEscolano, V. Morell-Giménez; Neural Networks 32: pp: 196-208 (2012), ISSN: 08936080; Impact Factor: 1.927 (JCR 2012) 74/79
  75. 75. Conclusions Publications Publications (II) • International conferences o “Point Light Source Estimation based on Scenes Recorded by a RGB-D camera” B. Boom, S. Orts-Escolano, X. Ning, S. McDonagh, P. Sandilands, R.B. Fisher; British Machine Vision Conference, BMVC 2013, Bristol, UK. Rank B o “Point Cloud Data Filtering and Downsampling using Growing Neural Gas” S. Orts-Escolano, V. Morell, J. Garcia-Rodriguez and M. Cazorla; International Joint Conference on Neural Networks, IJCNN 2013, Dallas, Texas. Rank A o “Natural User Interfaces in Volume Visualisation Using Microsoft Kinect” A. Angelopoulou, J. García Rodríguez, A.Psarrou, M. Mentzelopoulos, B. Reddy, S. OrtsEscolano, J.A. Serra. International Conference on Image Analysis and Processing, ICIAP2013, Naples, Italy: 11-19. Rank B o “Improving Drug Discovery using a neural networks based parallel scoring functions” H. Perez-Sanchez, G. D. Guerrero, J. M. Garcia, J. Pena, J. M. Cecilia, G. Cano, S. OrtsEscolano and J. Garcia-Rodriguez. International Joint Conference on Neural Networks, IJCNN 2013, Dallas, Texas. Rank A 75/79
  76. 76. Conclusions Publications Publications (III) • International conferences o “Improving 3D Keypoint Detection from Noisy Data Using Growing Neural Gas” J. García Rodríguez, M. Cazorla, S. Orts-Escolano, V. Morell. International Work-Conference on Artificial Neural Networks, IWANN 2013, Puerto de la Cruz, Tenerife, Spain: 480-487. Rank B o “3D Hand Pose Estimation with Neural Networks” J. A. Serra, J. García Rodríguez, S. Orts-Escolano, J. M. García Chamizo, A. Angelopoulou, A. Psarrou, M. Mentzelopoulos, J. Montoyo-Bojo, E. Domínguez. International WorkConference on Artificial Neural Networks, IWANN 2013, Puerto de la Cruz, Tenerife, Spain: 504-512. Rank B o “3D Gesture Recognition with Growing Neural Gas” J. A. Serra-Perez, J. Garcia-Rodriguez, S. Orts-Escolano, J. M. Garcia-Chamizo, A. Angelopoulou, A. Psarrou, M. Mentzeopoulos, J. Montoyo Bojo. International Joint Conference on Neural Networks. IJCNN 2013, Dallas, Texas. Rank A o “Multi-GPU based camera network system keeps privacy using Growing Neural Gas” S. Orts-Escolano, J. García Rodríguez, V. Morell, J.Azorín López, J. M. García Chamizo. International Joint Conference on Neural Networks (IJCNN) 2012, Brisbane, Australia, June: 1-8. Rank A 76/79
  77. 77. Conclusions Publications Publications (IV) • International conferences o “A study of registration techniques for 6DoF SLAM” V. Morell, M.Cazorla, D. Viejo, S. Orts-Escolano, J. García Rodríguez. International Conference of the Catalan Association for Artificial Intelligence, CCIA 2012, University of Alacant, Spain: 111-120. Rank B o “Fast Autonomous Growing Neural Gas” J. García Rodríguez, A. Angelopoulou, J. M. García Chamizo, A. Psarrou, S. Orts, V. Morell. International Joint Conference on Neural Networks, IJCNN 2011, San Jose, California: 725-732. Rank A o “Fast Image Representation with GPU-Based Growing Neural Gas” J.García Rodríguez, A. Angelopoulou, V. Morell, S. Orts, A. Psarrou, J. M. García Chamizo. International Work-Conference on Artificial Neural Networks, IWANN 2011, Torremolinos-Málaga, Spain: 58-65. Rank B o “Video and Image Processing with Self-Organizing Neural Networks” J. García Rodríguez, E. Domínguez, A. Angelopoulou, A. Psarrou, F. J. Mora-Gimeno, S. Orts, J. M. García Chamizo. International Work-Conference on Artificial Neural Networks, IWANN 2011, Torremolinos-Málaga, Spain: 98-104. Rank B 77/79
  78. 78. Conclusions Publications Publications (V) • National conferences o “Procesamiento de múltiples flujos de datos con Growing Neural Gas sobre Multi-GPU” S. Orts-Escolano, J. García-Rodríguez, V. Morell-Giménez. Jornadas de Paralelismo JP, Elche, España, 2012 • Book chapters o “A Review of Registration Methods on Mobile Robots” V. Morell-Gimenez, S. Orts-Escolano, J. García Rodríguez, M. Cazorla, D. Viejo. Robotic Vision: Technologies for Machine Learning and Vision Applications. IGI GLOBAL o “Computer Vision Applications of Self-Organizing Neural Networks” J. García-Rodríguez, J. M. García-Chamizo, S. Orts-Escolano, V. Morell-Gimenez, J.Serra-Perez, A. Angelolopoulou, M. Cazorla, D. Viejo. Robotic Vision: Technologies for Machine Learning and Vision Applications. IGI GLOBAL • Poster presentations o “6DoF pose estimation using Growing Neural Gas Network” S.Orts, J. Garcia-Rodriguez, D. Viejo, M. Cazorla, V. Morell, J. Serra. 5th International Conference on Cognitive Systems, Cogsys 2012, TU Vienna, Austria o “GPU Accelerated Growing Neural Gas Network” S. Orts, J. Garcia, V.Morell. Programming and Tuning Massively Parallel Systems, PUMPS 2011,Barcelona, Spain. (Honorable Mention by NVIDIA) 78/79
  79. 79. This presentation is licensed under a Creative Commons AttributionNonCommercial-ShareAlike 4.0 International License. 79/79
  80. 80. A Three-Dimensional Representation method for Noisy Point Clouds based on Growing Self-Organizing Maps accelerated on GPUs Author: Sergio Orts Escolano Supervisors: Dr. José García Rodríguez Dr. Miguel Ángel Cazorla Quevedo Doctoral programme in technologies for the information society

×