The document discusses real-time 3D reconstruction using depth cameras. It describes a research goal of understanding depth map refinement and hole-filling methods. A proposed algorithm first acquires depth data, then performs a hole-filling scan column-by-column before evaluating depth values. Experimental results found the technique improved surface quality over raw depth maps and achieved real-time reconstruction rates of up to 11 frames per second.

1. Real-time 3D reconstruction using depth cameras for
augmented teleoperation
Witenberg Santiago Rodrigues Souza(Undergraduate student)
Dr. Joohee Kim (Advisor)
Department of Electrical and Computer Engineering, Illinois Institute of Technology, Chicago, IL 60616, USA
wsantiag@hawk.iit.edu, joohee@ece.iit.edu
Research Goal
The research goal is understand real-time 3D
reconstruction using depth cameras:
Raw depth maps
Depth refinement
Hole-filling methods
Ground truth data set
Depth column based algorithm
Problem Description
Conclusion
Depth maps are noisy and holes.
Most common filter cannot solve the
problem for holes.
Transparent objects cannot be rendered.
Reflective surfaces return less depth values.
The process need to be parallelized to reduce
time consumption.
IIT Armour R&D Expo
Background
Depth refinement
Infra-Red sensor only
Refines raw depth maps according to
depth values and pixel distance.
Smoothing and hole-filing are difficult to
be successfully performed.
Shape deformation may happen.
Infra-Red sensor and RGB camera
Depth refinement is performed based on
depth maps and RGB images.
Hole-filling is performed without shape
deformation.
The system is dependent on illumination.
Misalignment between depth and color
information is an issue.
Proposed Algorithm
3D Reconstruction Architecture
Main steps:
1. Data acquisition
2. Hole-filling scan
3. Depth evaluation
Experimental Results
 Real-time 3D reconstruction technique has
been learned.
A proposed method has improved surface
quality by depth map refinement.
.
 Performance evaluation
[1] Pirovano, M. (2011/2012) “Kinfu – an open source implementation of Kinect Fusion+ case
study: implementing a 3D scanner with PCL” report
Hole-filling scan
The scanning is performed column by
column.
Once a null value is found, its 4-neighborhood
pixels are analyzed. If the 4-neighborhood
pixels are not sufficient to replace a null value
correctly the 8-neighborhood pixels are also
verified:
Di,j = (iN4)
or
(iN8 )
(a) (b) (c)
(d) (e)
(f)
(a) Original image, (b) corresponding depth image, (c)
enhanced depth map, (d) depth values for a row on a raw
depth map(e) same row after enhancement, (f) Original
values were kept and discontinuity was corrected.
A second evaluation is done by calculating the distance
between depth values:
Real-time 3D reconstruction
The new enhanced depth map is used as an
input for PCL Kinfu.[1]
Kinfu is able to read either live data from
sensor or off line dataset.
Method Absolute Trajectory
Error (ATE) [m]
Relative Pose
Error (RPE) [m]
Hole-filling by Column 0.0008 0.001
Step Time rate [fps]
Depth map enhancement 7
Kinfu offline raw data 4
Kinfu
Enhanced data 4
Live data 11
 Time Rate
 Performance comparison
 3D scene with the original depth map
 3D Scene with the new depth map
Data acquisition
ATE = Qi
-1
SPi
Where:
Q: ground truth, S: rigid body transformation, P: estimated trajectory
RPE = (Qi
-1
Qi+D )-1
(Pi
-1
Pi+D)
Where:
D = a fixed time interval
If ||D(i-1,j) - D(i-1,j-1)|| < || D(i,j-1) - D(i-1,j-1) || then
D(i, j)  D(i-1,j), with D(i-1,j) not null.
Quality- Raw Method
Surface jagged smoother
Objects poor poor/border issues
Quantity- Raw Method
Error - low