Paper and videos: http://www.socialstreetview.com This paper presents an immersive geo-spatial social media system for virtual and augmented reality environments. With the rapid growth of photo-sharing social media sites such as Flickr, Pinterest, and Instagram, geo-tagged photographs are now ubiquitous. However, the current systems for their navigation are unsatisfyingly one- or two-dimensional. In this paper, we present our prototype system, Social Street View, which renders the geo-tagged social media in its natural geo-spatial context provided by immersive maps, such as Google Street View. This paper presents new algorithms for fusing and laying out the social media in an aesthetically pleasing manner with geospatial renderings, validates them with respect to visual saliency metrics, suggests spatio-temporal filters, and presents a system architecture that is able to stream geo-tagged social media and render it across a range of display platforms spanning tablets, desktops, head-mounted displays, and large-area room-sized curved tiled displays. The paper concludes by exploring several potential use cases including immersive social storytelling, learning about culture and crowd-sourced tourism.

1. Social Street View: Blending Immersive Street Views
with Geo-tagged Social Media
Ruofei Du and Amitabh Varshney
The Augmentarium| UMIACS | University of Maryland, College Park
{ruofei, varshney} @ cs.umd.edu
www.SocialStreetView.com

2. image courtesy: plannedparenthood.org
Introduction
Social Media

3. image courtesy: adweek.com
Introduction
Social Media - Interactivity

4. image courtesy: forbes.com
Introduction
Social Media - Versatility

5. image courtesy: huffingtonpost.com
Introduction
Social Media - Social Relevance

7. EXIF
image courtesy: B&T Magazine

8. Introduction
Meta data - Time of Creation
image courtesy: Brian Clegg

9. Introduction
Meta data - Location of Creation
image courtesy: squarespace.com

10. Introduction
Meta data - Camera Parameters
image courtesy: naldzgraphics.net

11. image courtesy:
instagram.com,
facebook.com,
twitter.com
Related Work
Linear narrative visualization

12. image courtesy:
pinterest.com
Related Work
2D layout

13. Related Work
Natural Immersive Virtual Reality?
image courtesy: drivenhealthy.com

14. Related Work
Karnath et al. and Loomis et al.

15. Related Work
Karnath et al. and Loomis et al.

16. Related Work
Karnath et al. and Loomis et al.

17. Related Work
Visualization of Geo-tagged Information

18. Related Work
Visualization of Geo-tagged Information

19. Related Work
Visualization of Geo-tagged Information

20. Related Work
Visualization of Geo-tagged Information

21. Related Work
Visualization of Geo-tagged Information

22. image courtesy: panoramio.com
Related Work
Panoramio - Photos of the World
Cuenca and Manchón, 2005 - Now

23. image courtesy: twitterstand.umiacs.umd.edu
Related Work
TwitterStand: News in Tweets
Sankaranarayanan et al. SIGGIS 2009

24. image courtesy: flickr.com
Related Work
Flickr - World Map
Serdyukov et al. SIGIR 2009

25. image courtesy: photostand.umiacs.umd.edu
Related Work
PhotoStand for News Photos
Samet et al. VLDB 2013

26. Related Work
Visualization of Geo-tagged Information
(cont.)

27. Related Work
Visualization of Geo-tagged Information
(cont.)

28. Related Work
Visualization of Geo-tagged Information
(cont.)

29. Related Work
Visualization of Geo-tagged Information
(cont.)

30. Related Work
Visualization of Geo-tagged Information
(cont.)

31. Related Work
Visualization of Geo-tagged Information
(cont.)

32. Our Approach?

33. Social Street View

34. Social Street View

35. Demonstration
The Augmentarium, UMIACS
6000 x 3000 pixels

37. Conception, architecting & implementation
Social Street View
A mixed reality system that can depict geo-tagged
social media in immersive 3D web environments

38. Blending multiple modalities of
Street View + Social Media
Depth maps, normal maps, and road orientation
GPS coordinates and time creation

39. Enhancing visual augmentation
Maximal Poisson-disk sampling
Evaluated by image saliency metrics

40. Achieving cross-platform compatibility by
WebGL + Three.js
smartphones, tablets, desktop, high-resolution
large-area wide field of view tiled display walls, as
well as head-mounted displays.

41. Technical Challenges?

42. System Overview

43. Architecture
Social Street View System Flowchart

44. image courtesy: Google

45. Street View Cars - Cameras, Lasers and GPS
Image courtesy from Google Street View
image courtesy: Google

46. image courtesy: Google

47. Tiles of Panoramic Images
Image courtesy from Google Street View
image courtesy: Google

49. Panorama

51. Depth Map
Decoded from Google Maps API v3 and GSVPanoDepth.js

53. Normal Map
Decoded from Google Maps API v3

55. Road Orientations
Decoded from Google Maps API v3

56. image courtesy: Instagram

57. Multi-level Strategy for Streaming Social Media
Image courtesy from Instagram users in public domains
image courtesy: Instagram

58. SQL Database
Indexed by B+ Tree
logarithmic insertion & query

59. Bipartite Graph
G = V(V, S, E)

60. Haversine Formula
Andrew, 1805

61. Interface

63. Baseline
Random Uniform Sampling

64. Without uniform
sampling
Accumulation

65. Baseline
Random Uniform Sampling

66. Not preferred
Overlays high saliency regions

67. Add depth map
Remove sky and ground (most)

68. Add depth map
Remove sky and ground (most)

69. Can we ensure each image
aligns with the building geometries?

70. Add normal map
Remove all ground, align images

72. Can we reduce visual clutter and occlusion?

73. Maximal Poisson-disk Sampling
Gamito et al. Remove visual clutter and occlusion

74. Dart-throwing Algorithm
PixelPie by Ip et al. using vertex and fragment shaders
image courtesy:

75. Pixel-Pie Algorithm
Remove when occlusion occurs

76. Project Social Media Pictures
By Maximal Poisson-disc Sampling

77. Sampling Comparison
Remove visual clutter and occlusion

78. Scenic Landscapes
Using orientation of the road

79. Scenic Landscapes
Using orientation of the road

80. Scenic Landscapes
Using orientation of the road

82. Experiment
100 panoramas, 80,000+ social
media

83. Experiment
Nvidia Quadro K6000

84. Experiment
Conditions of panoramic images
Immersive high-resolution screens
Common Consumer-level Displays

85. Initialization Time
Panorama takes a while to load

86. After Prefetching
¾ - ⅘ time reduced

87. Rendering Time
Almost 58~60 FPS

88. Saliency Map
Hou et al. TPAMI 2011

89. Social Media
Coverage
100 panoramas for each algorithm

90. Social Media
Coverage
100 panoramas for each algorithm

91. Social Media
Coverage
100 panoramas for each algorithm

92. Potential Applications?

94. Stuck in traffic on our way to
Cabo with this awesome view
#roadtrip #cabo #view #mexico
Daniela on Instagram
July 12, 2014

97. Business Advertising
Museum, restaurant, real-estate ...

98. ... dinner started off with
amazing oysters paired with my
favorite Ruinart blanc de
blancs champagne
By frankiextah on
Instagram

99. Business Advertising
Museum, restaurant ...

100. Learning Culture
Taierzhuang, Chinese Spring Festivals

101. Crowd-sourced Tourism

102. Crowd-sourced Tourism

103. What if …
Temporal information is used for filtering and rendering?
image courtesy: mindtheproduct.com

104. Visual Enhancement
Spatio-Temporal Augmented Reality
image courtesy: webneel.com

106. Seasonal Effect
Winter

107. Seasonal Effect
Winter

108. Seasonal Effect
Spring

109. Seasonal Effect
Autumn in Paris

110. Failure Cases
A Square in London

111. Future Work
Digital City by 3D Reconstruction, Depth
Fusion and Augmented Reality
image courtesy:
wallpapervortex.com

112. Future Work
Incorperate with Open3D in Web3D 2016?
image courtesy: Lu et al.

113. Acknowledgement
Geollery | Shangfu Peng and Ruofei Du
www.geollery.com

114. Acknowledgement
Augmentarium Lab | GVIL | UMIACS

115. Acknowledgement
NSF | Nvidia | MPower | UMIACS

116. Video Fields
Web3D Sunday 9:20 - 9:40 AM

117. Social Street View
www.SocialStreetView.com
Thank you! Any questions or comments are welcome!
Ruofei Du and Amitabh Varshney
Augmentarium Lab | GVIL | UMIACS
Web3D 2016

118. Social Street View
Backup Slides

119. Future Work
Saliency
- Low-level image saliency for layout
- Real-time or post-processed saliency
maps

120. Future Work
User Study
- High-resolution large-area screens
- Head-mounted displays

121. Future Work
3D Reconstruction and AR
- Real-time geometric fusion
- HoloLens

122. Future Work
3D Reconstruction and AR
- What is the (experimentally-justifiable) motivation
for putting generic images into scenes?
- What is the relevance of which photos are
displayed? And how this affects the worth?
- Whether it is possible to try and improve this
relevance using vision techniques?

123. What is ignored in current social media sites?