Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

2,172 views

Published on

Paper and videos: http://www.socialstreetview.com

This paper presents an immersive geo-spatial social media system for virtual and augmented reality environments. With the rapid growth of photo-sharing social media sites such as Flickr, Pinterest, and Instagram, geo-tagged photographs are now ubiquitous. However, the current systems for their navigation are unsatisfyingly one- or two-dimensional. In this paper, we present our prototype system, Social Street View, which renders the geo-tagged social media in its natural geo-spatial context provided by immersive maps, such as Google Street View. This paper presents new algorithms for fusing and laying out the social media in an aesthetically pleasing manner with geospatial renderings, validates them with respect to visual saliency metrics, suggests spatio-temporal filters, and presents a system architecture that is able to stream geo-tagged social media and render it across a range of display platforms spanning tablets, desktops, head-mounted displays, and large-area room-sized curved tiled displays. The paper concludes by exploring several potential use cases including immersive social storytelling, learning about culture and crowd-sourced tourism.

Published in: Technology

Social Street View: Blending Immersive Street Views with Geo-tagged Social Media

  1. 1. Social Street View: Blending Immersive Street Views with Geo-tagged Social Media Ruofei Du and Amitabh Varshney The Augmentarium| UMIACS | University of Maryland, College Park {ruofei, varshney} @ cs.umd.edu www.SocialStreetView.com
  2. 2. image courtesy: plannedparenthood.org Introduction Social Media
  3. 3. image courtesy: adweek.com Introduction Social Media - Interactivity
  4. 4. image courtesy: forbes.com Introduction Social Media - Versatility
  5. 5. image courtesy: huffingtonpost.com Introduction Social Media - Social Relevance
  6. 6. EXIF image courtesy: B&T Magazine
  7. 7. Introduction Meta data - Time of Creation image courtesy: Brian Clegg
  8. 8. Introduction Meta data - Location of Creation image courtesy: squarespace.com
  9. 9. Introduction Meta data - Camera Parameters image courtesy: naldzgraphics.net
  10. 10. image courtesy: instagram.com, facebook.com, twitter.com Related Work Linear narrative visualization
  11. 11. image courtesy: pinterest.com Related Work 2D layout
  12. 12. Related Work Natural Immersive Virtual Reality? image courtesy: drivenhealthy.com
  13. 13. Related Work Karnath et al. and Loomis et al.
  14. 14. Related Work Karnath et al. and Loomis et al.
  15. 15. Related Work Karnath et al. and Loomis et al.
  16. 16. Related Work Visualization of Geo-tagged Information
  17. 17. Related Work Visualization of Geo-tagged Information
  18. 18. Related Work Visualization of Geo-tagged Information
  19. 19. Related Work Visualization of Geo-tagged Information
  20. 20. Related Work Visualization of Geo-tagged Information
  21. 21. image courtesy: panoramio.com Related Work Panoramio - Photos of the World Cuenca and Manchón, 2005 - Now
  22. 22. image courtesy: twitterstand.umiacs.umd.edu Related Work TwitterStand: News in Tweets Sankaranarayanan et al. SIGGIS 2009
  23. 23. image courtesy: flickr.com Related Work Flickr - World Map Serdyukov et al. SIGIR 2009
  24. 24. image courtesy: photostand.umiacs.umd.edu Related Work PhotoStand for News Photos Samet et al. VLDB 2013
  25. 25. Related Work Visualization of Geo-tagged Information (cont.)
  26. 26. Related Work Visualization of Geo-tagged Information (cont.)
  27. 27. Related Work Visualization of Geo-tagged Information (cont.)
  28. 28. Related Work Visualization of Geo-tagged Information (cont.)
  29. 29. Related Work Visualization of Geo-tagged Information (cont.)
  30. 30. Related Work Visualization of Geo-tagged Information (cont.)
  31. 31. Our Approach?
  32. 32. Social Street View
  33. 33. Social Street View
  34. 34. Demonstration The Augmentarium, UMIACS 6000 x 3000 pixels
  35. 35. Conception, architecting & implementation Social Street View A mixed reality system that can depict geo-tagged social media in immersive 3D web environments
  36. 36. Blending multiple modalities of Street View + Social Media Depth maps, normal maps, and road orientation GPS coordinates and time creation
  37. 37. Enhancing visual augmentation Maximal Poisson-disk sampling Evaluated by image saliency metrics
  38. 38. Achieving cross-platform compatibility by WebGL + Three.js smartphones, tablets, desktop, high-resolution large-area wide field of view tiled display walls, as well as head-mounted displays.
  39. 39. Technical Challenges?
  40. 40. System Overview
  41. 41. Architecture Social Street View System Flowchart
  42. 42. image courtesy: Google
  43. 43. Street View Cars - Cameras, Lasers and GPS Image courtesy from Google Street View image courtesy: Google
  44. 44. image courtesy: Google
  45. 45. Tiles of Panoramic Images Image courtesy from Google Street View image courtesy: Google
  46. 46. Panorama
  47. 47. Depth Map Decoded from Google Maps API v3 and GSVPanoDepth.js
  48. 48. Normal Map Decoded from Google Maps API v3
  49. 49. Road Orientations Decoded from Google Maps API v3
  50. 50. image courtesy: Instagram
  51. 51. Multi-level Strategy for Streaming Social Media Image courtesy from Instagram users in public domains image courtesy: Instagram
  52. 52. SQL Database Indexed by B+ Tree logarithmic insertion & query
  53. 53. Bipartite Graph G = V(V, S, E)
  54. 54. Haversine Formula Andrew, 1805
  55. 55. Interface
  56. 56. Baseline Random Uniform Sampling
  57. 57. Without uniform sampling Accumulation
  58. 58. Baseline Random Uniform Sampling
  59. 59. Not preferred Overlays high saliency regions
  60. 60. Add depth map Remove sky and ground (most)
  61. 61. Add depth map Remove sky and ground (most)
  62. 62. Can we ensure each image aligns with the building geometries?
  63. 63. Add normal map Remove all ground, align images
  64. 64. Can we reduce visual clutter and occlusion?
  65. 65. Maximal Poisson-disk Sampling Gamito et al. Remove visual clutter and occlusion
  66. 66. Dart-throwing Algorithm PixelPie by Ip et al. using vertex and fragment shaders image courtesy:
  67. 67. Pixel-Pie Algorithm Remove when occlusion occurs
  68. 68. Project Social Media Pictures By Maximal Poisson-disc Sampling
  69. 69. Sampling Comparison Remove visual clutter and occlusion
  70. 70. Scenic Landscapes Using orientation of the road
  71. 71. Scenic Landscapes Using orientation of the road
  72. 72. Scenic Landscapes Using orientation of the road
  73. 73. Experiment 100 panoramas, 80,000+ social media
  74. 74. Experiment Nvidia Quadro K6000
  75. 75. Experiment Conditions of panoramic images Immersive high-resolution screens Common Consumer-level Displays
  76. 76. Initialization Time Panorama takes a while to load
  77. 77. After Prefetching ¾ - ⅘ time reduced
  78. 78. Rendering Time Almost 58~60 FPS
  79. 79. Saliency Map Hou et al. TPAMI 2011
  80. 80. Social Media Coverage 100 panoramas for each algorithm
  81. 81. Social Media Coverage 100 panoramas for each algorithm
  82. 82. Social Media Coverage 100 panoramas for each algorithm
  83. 83. Potential Applications?
  84. 84. Stuck in traffic on our way to Cabo with this awesome view #roadtrip #cabo #view #mexico Daniela on Instagram July 12, 2014
  85. 85. Business Advertising Museum, restaurant, real-estate ...
  86. 86. ... dinner started off with amazing oysters paired with my favorite Ruinart blanc de blancs champagne By frankiextah on Instagram
  87. 87. Business Advertising Museum, restaurant ...
  88. 88. Learning Culture Taierzhuang, Chinese Spring Festivals
  89. 89. Crowd-sourced Tourism
  90. 90. Crowd-sourced Tourism
  91. 91. What if … Temporal information is used for filtering and rendering? image courtesy: mindtheproduct.com
  92. 92. Visual Enhancement Spatio-Temporal Augmented Reality image courtesy: webneel.com
  93. 93. Seasonal Effect Winter
  94. 94. Seasonal Effect Winter
  95. 95. Seasonal Effect Spring
  96. 96. Seasonal Effect Autumn in Paris
  97. 97. Failure Cases A Square in London
  98. 98. Future Work Digital City by 3D Reconstruction, Depth Fusion and Augmented Reality image courtesy: wallpapervortex.com
  99. 99. Future Work Incorperate with Open3D in Web3D 2016? image courtesy: Lu et al.
  100. 100. Acknowledgement Geollery | Shangfu Peng and Ruofei Du www.geollery.com
  101. 101. Acknowledgement Augmentarium Lab | GVIL | UMIACS
  102. 102. Acknowledgement NSF | Nvidia | MPower | UMIACS
  103. 103. Video Fields Web3D Sunday 9:20 - 9:40 AM
  104. 104. Social Street View www.SocialStreetView.com Thank you! Any questions or comments are welcome! Ruofei Du and Amitabh Varshney Augmentarium Lab | GVIL | UMIACS Web3D 2016
  105. 105. Social Street View Backup Slides
  106. 106. Future Work Saliency - Low-level image saliency for layout - Real-time or post-processed saliency maps
  107. 107. Future Work User Study - High-resolution large-area screens - Head-mounted displays
  108. 108. Future Work 3D Reconstruction and AR - Real-time geometric fusion - HoloLens
  109. 109. Future Work 3D Reconstruction and AR - What is the (experimentally-justifiable) motivation for putting generic images into scenes? - What is the relevance of which photos are displayed? And how this affects the worth? - Whether it is possible to try and improve this relevance using vision techniques?
  110. 110. What is ignored in current social media sites?

×