Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Video Fields: Fusing Multiple Surveillance Videos into a
Dynamic Virtual Environment
Ruofei Du, Sujal Bista, Amitabh Varsh...
image courtesy: university of maryland, college park
Introduction
Surveillance Videos - Monitoring
image courtesy: www.icsc.org
Introduction
Surveillance Videos – Shopping Centers
image courtesy: wikipedia
Introduction
Surveillance Videos - Airports
image courtesy: wikipedia
Introduction
Surveillance Videos – Train stations
image courtesy: university of maryland, college park
Introduction
Surveillance Videos - Campuses
image courtesy: university of maryland, college park
Introduction
Surveillance Videos - Conventional
image courtesy: theimaginativeconservative.org
Introduction
Surveillance Videos – Cognitive Burden
image courtesy: university of maryland, college park
Introduction
Surveillance Videos – Fusing & Interpreting
Related Work
Fusing Multiple Static Photographs
Related Work
Fusing Multiple Static Photographs
Related Work
Fusing Multiple Static Photographs
Related Work
Fusing Multiple Static Photographs
Related Work
Fusing Multiple Static Photographs
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
RGB
Related Work
Fusing Multiple Dynamic Videos
RGB
RGBD
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
SIGGRAPH 2016
Wednesday, 3:30-4:00 PM
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
Related Work
Fusing Multiple Dynamic Videos
Our Approach?
Video Fields
Video Fields
Introduction
Video Field
Introduction
Video Field
Conception, architecting & implementation
Video Fields
A mixed reality system that fuses multiple
surveillance videos into...
Integrating automatic segmentation of
moving entities
Video Fields Rendering
Real-time fragment shader processing
Two algorithms to fuse multiple videos
Early & deferred pruning
These methods use voxels and meshes respectively
to render...
Achieving cross-platform compatibility by
WebGL + Three.js
smartphones, tablets, desktop, high-resolution
large-area wide ...
System Overview
Architecture
Video Fields Flowchart
Architecture
Video Fields Flowchart
Architecture
Video Fields Flowchart
Architecture
Video Fields Flowchart
Background Modeling
Motivation
• Provide a background texture for each camera
• Identify moving entities in the rendering ...
Background Modeling
Gaussian Mixture Models (GMM)
Background Modeling
Advantages [Stauffer and Grimson]
More adaptive with:
• different lighting conditions,
• repetitive mo...
Architecture
Video Fields Flowchart
Segmentation
Moving Entities
Background Modeling
Gaussian Mixture Models (GMM)
Architecture
Video Fields Flowchart
Visibility Test
Plus Opacity Modulation
Architecture
Video Fields Flowchart
Video Fields Mapping
Overview
Video Fields Mapping
Challenges
1. Vertex in the 3D models -> Pixel in the texture space
2. Pixel in the texture space -> ...
Video Fields Mapping
Projection Mapping
Video Fields Mapping
Perspective correction
Video Fields Mapping
Depth Map / Hashing Function
Early Pruning for Rendering
Moving Entities
Voxels
Deferred Pruning for
Rendering Moving Entities
Billboards
Visual Comparison
Early Pruning vs. Deferred Pruning
View-dependent Rendering
View-dependent Rendering
View-dependent Rendering
View-dependent Rendering
Experimental Results
Early Pruning vs. Deferred Pruning
Experimental Results
Early Pruning vs. Deferred Pruning
Experimental Results
Early Pruning vs. Deferred Pruning
Visual Comparison
Early Pruning vs. Deferred Pruning
Future Work
Scale Up - Hundreds of cameras
Future Work
Bandwidth Problem
Future Work
Holoportation with RGB cameras
Acknowledgement
Augmentarium Lab | GVIL | UMIACS
Acknowledgement
NSF | Nvidia | MPower | UMIACS
Video Fields
www.Video-Fields.com
Thank you! Questions or comments?
Ruofei Du and Amitabh Varshney
Augmentarium Lab | GVIL...
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
Upcoming SlideShare
Loading in …5
×

Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment

2,705 views

Published on

Paper and videos: http://www.videofields.com
Another interesting project: http://socialstreetview.com

Video Fields system fuses multiple videos, camera-world matrices from a calibration interface, static 3D models, as well as satellite imagery into a novel dynamic virtual environment. Video Fields integrates automatic segmentation of moving entities during the rendering pass and achieves view-dependent rendering in two ways: early pruning and deferred pruning. Video Fields takes advantage of the WebGL and WebVR technology to achieve cross-platform compatibility across smart phones, tablets, desktops, high-resolution tiled curved displays, as well as virtual reality head-mounted displays. See the supplementary video at http://video-fields.com.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment

  1. 1. Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment Ruofei Du, Sujal Bista, Amitabh Varshney The Augmentarium| UMIACS | University of Maryland, College Park {ruofei, sujal, varshney} @ cs.umd.edu www.VideoFields.com
  2. 2. image courtesy: university of maryland, college park Introduction Surveillance Videos - Monitoring
  3. 3. image courtesy: www.icsc.org Introduction Surveillance Videos – Shopping Centers
  4. 4. image courtesy: wikipedia Introduction Surveillance Videos - Airports
  5. 5. image courtesy: wikipedia Introduction Surveillance Videos – Train stations
  6. 6. image courtesy: university of maryland, college park Introduction Surveillance Videos - Campuses
  7. 7. image courtesy: university of maryland, college park Introduction Surveillance Videos - Conventional
  8. 8. image courtesy: theimaginativeconservative.org Introduction Surveillance Videos – Cognitive Burden
  9. 9. image courtesy: university of maryland, college park Introduction Surveillance Videos – Fusing & Interpreting
  10. 10. Related Work Fusing Multiple Static Photographs
  11. 11. Related Work Fusing Multiple Static Photographs
  12. 12. Related Work Fusing Multiple Static Photographs
  13. 13. Related Work Fusing Multiple Static Photographs
  14. 14. Related Work Fusing Multiple Static Photographs
  15. 15. Related Work Fusing Multiple Dynamic Videos
  16. 16. Related Work Fusing Multiple Dynamic Videos RGB
  17. 17. Related Work Fusing Multiple Dynamic Videos RGB RGBD
  18. 18. Related Work Fusing Multiple Dynamic Videos
  19. 19. Related Work Fusing Multiple Dynamic Videos
  20. 20. Related Work Fusing Multiple Dynamic Videos
  21. 21. Related Work Fusing Multiple Dynamic Videos
  22. 22. Related Work Fusing Multiple Dynamic Videos
  23. 23. Related Work Fusing Multiple Dynamic Videos
  24. 24. Related Work Fusing Multiple Dynamic Videos
  25. 25. Related Work Fusing Multiple Dynamic Videos
  26. 26. Related Work Fusing Multiple Dynamic Videos SIGGRAPH 2016 Wednesday, 3:30-4:00 PM
  27. 27. Related Work Fusing Multiple Dynamic Videos
  28. 28. Related Work Fusing Multiple Dynamic Videos
  29. 29. Related Work Fusing Multiple Dynamic Videos
  30. 30. Our Approach?
  31. 31. Video Fields
  32. 32. Video Fields
  33. 33. Introduction Video Field
  34. 34. Introduction Video Field
  35. 35. Conception, architecting & implementation Video Fields A mixed reality system that fuses multiple surveillance videos into an immersive virtual environment,
  36. 36. Integrating automatic segmentation of moving entities Video Fields Rendering Real-time fragment shader processing
  37. 37. Two algorithms to fuse multiple videos Early & deferred pruning These methods use voxels and meshes respectively to render moving entities in the video fields
  38. 38. Achieving cross-platform compatibility by WebGL + Three.js smartphones, tablets, desktop, high-resolution large-area wide field of view tiled display walls, as well as head-mounted displays.
  39. 39. System Overview
  40. 40. Architecture Video Fields Flowchart
  41. 41. Architecture Video Fields Flowchart
  42. 42. Architecture Video Fields Flowchart
  43. 43. Architecture Video Fields Flowchart
  44. 44. Background Modeling Motivation • Provide a background texture for each camera • Identify moving entities in the rendering stage • Reduce the network bandwidth requirements
  45. 45. Background Modeling Gaussian Mixture Models (GMM)
  46. 46. Background Modeling Advantages [Stauffer and Grimson] More adaptive with: • different lighting conditions, • repetitive motions of scene elements, • moving entities in slow motion
  47. 47. Architecture Video Fields Flowchart
  48. 48. Segmentation Moving Entities
  49. 49. Background Modeling Gaussian Mixture Models (GMM)
  50. 50. Architecture Video Fields Flowchart
  51. 51. Visibility Test Plus Opacity Modulation
  52. 52. Architecture Video Fields Flowchart
  53. 53. Video Fields Mapping Overview
  54. 54. Video Fields Mapping Challenges 1. Vertex in the 3D models -> Pixel in the texture space 2. Pixel in the texture space -> Vertex on the ground • The second is useful for projecting a 2D segmentation of a moving entity to the 3D world
  55. 55. Video Fields Mapping Projection Mapping
  56. 56. Video Fields Mapping Perspective correction
  57. 57. Video Fields Mapping Depth Map / Hashing Function
  58. 58. Early Pruning for Rendering Moving Entities Voxels
  59. 59. Deferred Pruning for Rendering Moving Entities Billboards
  60. 60. Visual Comparison Early Pruning vs. Deferred Pruning
  61. 61. View-dependent Rendering
  62. 62. View-dependent Rendering
  63. 63. View-dependent Rendering
  64. 64. View-dependent Rendering
  65. 65. Experimental Results Early Pruning vs. Deferred Pruning
  66. 66. Experimental Results Early Pruning vs. Deferred Pruning
  67. 67. Experimental Results Early Pruning vs. Deferred Pruning
  68. 68. Visual Comparison Early Pruning vs. Deferred Pruning
  69. 69. Future Work Scale Up - Hundreds of cameras
  70. 70. Future Work Bandwidth Problem
  71. 71. Future Work Holoportation with RGB cameras
  72. 72. Acknowledgement Augmentarium Lab | GVIL | UMIACS
  73. 73. Acknowledgement NSF | Nvidia | MPower | UMIACS
  74. 74. Video Fields www.Video-Fields.com Thank you! Questions or comments? Ruofei Du and Amitabh Varshney Augmentarium Lab | GVIL | UMIACS Web3D 2016

×