Graphics hardware has evolved from specialized graphics processors to general purpose GPUs capable of parallel data processing. This document outlines several ways GPUs can be used for visual computing and general data processing tasks. It describes using GPUs for classical computer vision tasks like stereo reconstruction as well as general data structures like quadtrees, octrees, and data compaction/expansion algorithms. The key challenges of parallelizing algorithms on GPUs and techniques like histogram pyramids for data-parallel processing of feature streams are discussed. Several publications applying these GPU techniques to problems in lighting simulation, point cloud generation, and iso-surface extraction are also summarized.

1. GPU Data Structures
for Graphics and Vision
Promotionskolloquium, May 6th 2011
Gernot Ziegler
Dept. of Computer Graphics
(3D Video and Vision-Based Graphics Group)

2. Outline
Graphics Hardware:
Original Purpose and Recent Development
Classical Usage in Visual Computing
 Free Viewpoint Video Compression
 Color and Depth Reprojection
 Hierarchical Image Processing
General Data Processing
 Data Compaction with the HistoPyramid
 Quadtree and Octree Generation
 Data Expansion with the HistoPyramid
Conclusion

3. Graphics Hardware: Original Purpose
 Graphics hardware accelerates typical data operations of
computer graphics (pixel moves, triangle rasterization)
 GPU is simpler in design than CPU, but massively parallel.

4. Graphics Hardware: Capabilities
 ~2003: Graphics Hardware becomes programmable:
GPU (Graphics Processing Unit)

5. Graphics Hardware: Capabilities
 Data can now be anything (floating point & integer)
 General Purpose Computing on GPU = GPGPU
"Classical Usage" in Visual Computing (still graphics-related)
– Computer Vision
– Video processing
– Volume analysis
General Data Processing
– PDE / ODE solver
– Spatial Data Structure Generation
– Database Ops
– Etc…
Game of Life
(Early GPGPU by S. Green)

6. Classical Usage in Visual Computing

7. Free Viewpoint Video Compression
(Chapter 3)
 Map video footage into texture domain via proxy 3D model

8. Free Viewpoint Video Compression
(Chapter 3)
 Obtain texture surface masking via shadow mapping

9. Free Viewpoint Video Compression: Publications
 G. Ziegler, H. Lensch, N. Ahmed, M. Magnor, H.-P. Seidel.
Multi-Video Compression in Texture Space.
11th IEEE Intl Conference on Image Processing (ICIP 2004),
Singapore, pp. 2467-2470, 2004.
 G. Ziegler, H. Lensch, M. Magnor, H.-P. Seidel. Multi-Video
Compression in Texture Space using 4D SPIHT.
6th IEEE Workshop on Multimedia Signal Processing, Siena,
Italy, pp. 39-42, 2004.

10. Color and Depth Reprojection
(Chapter 4)
 Depth-map "Projection" via proxy mesh & vertex shader
Novel View reconstruction
from partial depth camera views

11. Color and Depth Reprojection
(Chapter 4)
Blending by View Angle Our per-pixel approach (Purple: Blended areas)

12. Hierarchical Image Processing: Stereo reconstruction
(Chapter 5.1)
 Projective texturing in plane-sweep (GPU feedback, coarse-to-fine)

13. Hierarchical Image Processing: Stereo reconstruction
(Chapter 5.1)
 Projective texturing in plane-sweep (GPU feedback, coarse-to-fine)

14. Hierarchical Image Processing: Reduction
(Chapter 5.2)
 Mipmap-like reduction: Dominant feature region, noise reduction

15. Hierarchical Image Processing: Reduction
(Thesis Chapter 5.3)
 Histogram of local gradients guides lens warp compensation

16. General Data Processing

17. Graphics Hardware: Capabilities
 GPU has massive computation and memory throughput

18. Graphics Hardware: Limitations
 GPU is connected with CPU via narrow databus
(bandwidth bottleneck, approx. 4 GB/s)
 GPU is a Stream processor:
– 10K thread workload necessary to keep 100s cores busy
(data parallelization!)
– Thread switching lightweight, but synchronization expensive!
– Each thread can only write at a fixed position
 Algorithms must be redesigned for GPU!

19. General Data Processing
Data Compaction
(Chapter 6)

20. Data-Parallel Algorithm Challenges
 Example from Computer Vision: List of all black pixels in an image
 Step 1: Detect black pixels:
 Step 2: Create a list of detected pixels

21. Previous approach to feature list generation
 Step 2 (List generation) was not possible on GPU!
 2a: GPU marks local features (e.g. thresholding, filtering)
 2b: CPU searches image and generates feature list
 But: Bus transfers expensive:
GPU useful only for complex feature isolation.
(e.g. large filter convolution & thresholding)

22. Our approach: Feature list generation on GPU
 We generate feature lists on the GPU using data compaction.
 Pixel/Voxels/Feature input is abstracted "data element stream”.
 Compaction keeps only elements deemed relevant for output.
1D example (keep all elements that are blue):
 Data flow:
Massive speedup due to strongly reduced bus dataflow!
1 1 0 1A B C D E F 1 1 1A B D E

23. Data Compaction: Problem task in 1D
 Keep number of elements from input, based on a Classifier:
 Implementation is trivial on CPU, single-thread.
 On GPU: Need to parallelize into 10k threads!
 First count number of output elements
using data-parallel reduction!

24. Data Compaction via HistoPyramid:Buildup
 First, count number of output elements,
e.g. 4:1 data-parallel reduction
 (Note the reduction pyramid, it is retained - HistoPyramid)
 Can now allocate compact output, no spill.
 But how are output elements generated?
Histogram pyramid /
HistoPyramid

25. Data Compaction via HistoPyramid: Traversal
 Output generate: Start one thread per output element
 Each output thread traverses reduction pyramid (read-only)
 No read/write hazards = Data-parallel output writing!
 As many threads as output elements

26. HistoPyramid: 2D Data Compaction
 1D was tutorial, actual implementation is 2D !
 Dataflow diagram:

27. GPU Data Compaction:Publications
 Data Compaction fast enough for real-time volume analysis
 First application: Mesh-to-volume-to-point cloud in real-time!
 G. Ziegler, A. Tevs, C. Theobalt, H.-P. Seidel
On-the-fly Point Clouds through Histogram Pyramids
11th International Fall Workshop on Vision, Modeling and
Visualization 2006 (VMV2006), 2006, pp. 137-144.

28. GPU Data Compaction:Publications
 Data Compaction fast enough for real-time volume analysis
 First application: Mesh-to-volume-to-point cloud in real-time!
 G. Ziegler, A. Tevs, C. Theobalt, H.-P. Seidel
On-the-fly Point Clouds through Histogram Pyramids
11th International Fall Workshop on Vision, Modeling and
Visualization 2006 (VMV2006), 2006, pp. 137-144.

29. GPU Data Compaction:Publications
 Vector Field Contours: View-dependent vectorfield analysis to
visualize contour lines throughout the volume
 Data Compaction delivers seedpoints for contour lines in ms!
 T. Annen, H. Theisel, C. Rössl, G. Ziegler, H.-P. Seidel
Vector Field Contours
Graphics Interface 2008, Windsor/Canada, 2008, pp. 97-105

30. General Data Processing
Quadtree and Octree Generation
(Chapter 8 and 9)

31. GPU Quadtrees: Introduction
 2D Reduction follows a quadtree-like reduction pattern.
 By tracking feature similarity in reduction,
quadtrees can be created from the reduction pyramid!

32. GPU QuadTree: Publications
 Speed (ms) enables real-time quadtree processing from video!
e.g. for Compression, Vision,..
 G. Ziegler, R. Dimitrov, C. Theobalt, H-P. Seidel.
Real-time Quadtree Analysis using HistoPyramids.
SPIE Electronic Imaging conference, San Jose/USA, 2007.

33. GPU Octree
(Chapter 9)
 Feature Clustering extended to 3D volumes
 Octrees from Volume Data
 New algorithm, pointer octrees
(e.g. for spatial data structures)
 Real-time creation of
high-resolution octrees
from meshes possible!

34. General Data Processing
Data Expansion
(Chapter 7)

35. Data Expansion via HistoPyramid:
Problem task
 We have a predicate function that determines
how many output copies to create from each input element:
 Implementation is trivial on CPU
 GPU: Input can be divided amongst threads, but:
Where shall each thread write its output?
 Insight:
HistoPyramid traversal works even here!

36. Data Expansion via HistoPyramid:
HP Buildup
 First, count number of output elements, e.g. via 4:1 reduction:

37. Data Expansion via HistoPyramid:
HP Traversal (single output copy)
 Traversal for single output elements:
 Exactly like data compaction, but: Mind local key index kL

38. Data Expansion via HistoPyramid:
HP Traversal (multiple output copies)
 Traversal for multiple output elements:
 kL determines number of copy. Still: one thread for each copy!

39. Data Expansion via HistoPyramid:
HP Traversal (multiple output copies)
 Traversal for multiple output elements:
 kL determines number of copy.
 Observation: Thread can modify input before write-out!
 Thus: Output can be modified version of input based on kL.
 e.g. Geometry Creation:
 (Generic algorithm…)

40. Data Expansion: Eikonal Rendering (Publication I)
 Compute light transport through volume objects of varying refraction
 Both real-time rendering and precomputed lighting simulation
 Lighting simulation requires adaptive light wavefront simulation
 I. Ihrke, G. Ziegler, A. Tevs, C. Theobalt, M. Magnor, H.-P. Seidel
Eikonal Rendering: Efficient Light Transport in Refractive Objects
ACM Transactions on Graphics 26 (3): 59-1 - 59-8, 2007
http://www.mpi-inf.mpg.de/resources/EikonalRendering/

41. Eikonal Rendering: Lighting Simulation
 For given light-object position, precompute lighting inside
the volumetric object for real-time novel view rendering.

42.  Lighting simulation implements numerical ODE solver on GPU.
 Subdivide light's wavefront into a set of patches
 Patch corners move as GPU particle system
– Each particle follows ray optics
 During update, some patches:
– weaken too much (discard)
– leave volume (discard)
– grow too large (tesselate)
 Since patch list is on GPU:
– Discard: Data Compaction
– Tesselate: Data Expansion
Eikonal Rendering: Wavefront Propagation

43. Eikonal Rendering: Short Demo

44. Eikonal Rendering: Short Demo

45. Data Expansion: Marching Cubes (Publication II)
 Marching Cubes algorithm extracts iso-surfaces from volumes
 Reformulate: Stream of voxels ...
– is first compacted to the relevant iso-surface voxels
– then expanded, becoming a stream of triangle vertices
 C. Dyken, G. Ziegler, C. Theobalt, and H.-P. Seidel
High-speed Marching Cubes using HistoPyramids
Computer Graphics Forum 27 (8): 2028-2039, 2008
http://www.sintef.no/hpmc

46. Performance of OpenGL approach (2007):
 Geometry shader (GS), e.g. NVIDIA GeForce 8, enabled
hardware data compaction & expansion for geometry -
should obsolete HistoPyramids, but HP-MC outperforms
geometry shader (HP-GS)!
HP-MC was 2007 fastest known MC algorithm.
(frames per second)

47. Conclusion and Outlook
(Chapter 10)

48. Conclusion and Outlook
 GPUs increasingly useful in general data processing
 Programming Model Restrictions not always bad
– Force programmer to change thought model
– E.g.: Fixed Output Location created HistoPyramid traversal concept
– Can be more efficient, even on more capable hardware!
(atomic counters, geometry shaders have less performance)
 Data-Parallel Algorithm Design is hard
– But once done, parallelizable over any number of available cores
(if sufficient data available)
– Hard to imagine that auto-parallelization can achieve this
 Future work
– Connected components, distance transforms, SATs…
– Accelerate further using CUDA C and OpenCL

49. Other work based on presented algorithms
Quadtree
 C. N. Vasconcelos, A. Sá, P. C. Carvalho, M. Gattass.
QuadN4tree: A GPU-Friendly Quadtree Leaves Neighborhood Structure.
Proc. of Computer Graphics International Conference (CGI) 2008.
 C. N. Vasconcelos, A. Sá, P. C. Carvalho, M. Gattass.
Using Quadtrees for Energy Minimization Via Graph Cuts.
Proc. of VMV - 12th Vision, Modeling, and Visualization Workshop, pp. 71-80.
Data Expansion
 C. Dyken, M. Reimers, J. Seland.
Real-time GPU Silhouette Refinement using adaptively blended Bézier patches.
Computer Graphics Forum, Volume 27, number 1, pp. 1-12, 2007.
Data Compaction (Implementation)
 J. Fung, S. Mann.
OpenVIDIA: parallel GPU computer vision.
Proc. of 13th annual ACM international conference on Multimedia, pp. 849 - 852.
http://openvidia.sf.net

50. End of Presentation

51. Recent Work

52. San Jose (CA) | September 23rd, 2010
Christopher Dyken, SINTEF Norway
Gernot Ziegler, NVIDIA UK
GPU-accelerated data expansion
for the Marching Cubes algorithm

53. HistoPyramid performance
 Accelerated HistoPyramids using CUDA C
 HistoPyramid BuildUp
— Reduce 5-to-1, but store only first four sums!
— Build several levels via on-GPU shared memory
(less video memory transactions)
 Marching Cubes specific
— Share scalar input data amongst neighbouring MC cells
(through shared memory)

54. Backpack (iso=0.4) (www.volvis.org)
Size 512x512x373 (187 mb)
Triangles 3 745 320 (0.039 tris/cell)
OpenGL HP4MC 13 fps (1291 mvps)
CUDA-OpenGL HP5MC 43 fps (4129 mvps)
Speedup
3.2x
Head aneuyrism (iso=0.4) (www.volvis.org)
Size 512x512x512 (256 mb)
Triangles 583 610 (0.004 tris/cell)
OpenGL HP4MC 15 fps (2034 mvps)
CUDA-OpenGL HP5MC 78 fps (10399 mvps)
Speedup
5.1x
Christmas tree (iso=0.05) (TU Wien)
Size 512x499x512 (250 mb)
Triangles 5 629 532 (0.043 tris/cell)
OpenGL HP4MC 10 fps (1358 mvps)
CUDA-OpenGL HP5MC 28 fps (3704 mvps)
Speedup
2.7x
5123-ish 16-bit performance

56. End of Presentation