Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media

35 views

Published on

Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.

Social media in virtual reality is in a high-growth market segment with influential products and services in virtual tourism, remote education, and business meetings. Nevertheless, previous systems have never achieved an online platform which renders a 6DoF mirrored world with geotagged social media in real time. In this paper, we introduce the technical detail behind Geollery.com which reconstructs a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local street views and depth maps. When viewed from afar, we apply projection mappings to 3D geometries extruded from building polygons for a coarse level of detail. In contrast to teleportation, our system allows users to virtually walk through the mirrored world at the street level. Our system integrates geotagged social media from both internal users and external sources such as Twitter, Yelp, and Flicker. We validate our real-time strategies of Geollery.com on various platforms including mobile phones, workstations, and head-mounted displays.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media

  1. 1. Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media Ruofei Du† , David Li† , and Amitabh Varshney {ruofei, dli7319, varshney}@umiacs.umd.edu | www.Geollery.com | Web3D 2019, Los Angeles, USA UMIACS THE AUGMENTARIUM VIRTUAL AND AUGMENTED REALITY LAB AT THE UNIVERSITY OF MARYLAND COMPUTER SCIENCE UNIVERSITY OF MARYLAND, COLLEGE PARK Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media
  2. 2. image courtesy: plannedparenthood.org Introduction Social Media 2
  3. 3. image courtesy: huffingtonpost.com Introduction Social Media + Topics 3
  4. 4. Motivation Social Media + XR 4
  5. 5. image courtesy: instagram.com, facebook.com, twitter.com Motivation Social Media + XR 5
  6. 6. Motivation 2D layout image courtesy: pinterest.com 6
  7. 7. Motivation Immersive Mixed Reality? image courtesy: viralized.com 7
  8. 8. Motivation Pros and cons of the classic 8
  9. 9. Motivation Pros and cons of the classic 9
  10. 10. Related Work 2D Geospatial Visualization 10
  11. 11. Related Work 2D Geospatial Visualization 11
  12. 12. Related Work 3D Geospatial Visualization 12
  13. 13. Related Work Social Street View, Du and Varshney Web3D 2016 Best Paper Award 13
  14. 14. Related Work Social Street View, Du and Varshney Web3D 2016 Best Paper Award 14
  15. 15. Related Work Social Street View, Du and Varshney Web3D 2016 Best Paper Award 15
  16. 16. Related Work Social Street View, Du and Varshney Web3D 2016 Best Paper Award 16
  17. 17. Related Work VirtualOulu: Toni Alatalo et al. Web3D 2016 17
  18. 18. Related Work Virtual Oulu, Kukka et al. CSCW 2017 18
  19. 19. Related Work 3D Visual Popularity Bulbul and Dahyot, 2017 19
  20. 20. Related Work Immersive Trip Reports Brejcha et al. UIST 2018 20
  21. 21. Related Work High Fidelity, Inc. 21
  22. 22. Related Work Facebook Spaces, 2017 22
  23. 23. Geollery A Mixed-Reality Social Media Platform, CHI 2019 23
  24. 24. Geollery.com A Mixed-Reality Social Media Platform 24
  25. 25. What's Next? Research Question ⅓ Du et al. Geollery, CHI 2019. What may a social media platform look like in mixed reality? 25
  26. 26. What's Next? Research Question ⅔ Du et al. Geollery, CHI 2019. What if we could allow social media sharing in a live mirrored world? 26
  27. 27. What's Next? Research Question 3/3 Du et al. Geollery, CHI 2019. What use cases can we benefit from social media platform in XR? 27
  28. 28. System Overview Geollery V1 Workflow 28
  29. 29. Design Space Geollery vs. Social Street View Du et al. Geollery, CHI 2019. 29
  30. 30. User Study Quantitative Evaluation Du et al. Geollery, CHI 2019. 30
  31. 31. Insights What we learned 31 High-quality content and seed users play key roles
  32. 32. Insights What we learned 32 Interactivity and panoramic textures increase immersion.
  33. 33. [I will use it for] exploring new places. If I am going on vacation somewhere, I could immerse myself into the location. If there are avatars around that area, I could ask questions. P1 / M 33
  34. 34. I think it (Geollery) will be useful for families. I just taught my grandpa how to use Facetime last week and it would great if I could teleport to their house and meet with them, then we could chat and share photos with our avatars. P2 / F 34
  35. 35. What if we could reconstruct a high-quality, all textured, walkable mirrored world with geotagged social media in real time? 35
  36. 36. Geollery.com v2: a major leap 36
  37. 37. System Overview Geollery Workflow 37
  38. 38. System Overview Geollery Workflow 38
  39. 39. System Overview Given (latitude, longitude) 39
  40. 40. System Overview 2D Map Data 40
  41. 41. System Overview 2D Map Data 41
  42. 42. System Overview +Avatar 42
  43. 43. System Overview +Avatar +Trees +Clouds 43
  44. 44. System Overview +Avatar +Trees +Clouds +Night 44
  45. 45. System Overview Street View Panoramas 45
  46. 46. System Overview Street View Panoramas 46
  47. 47. System Overview Street View Panoramas 47
  48. 48. System Overview Geollery Workflow 48
  49. 49. System Overview Geollery Workflow 49
  50. 50. System Overview Geollery Workflow 50
  51. 51. System Overview Geollery Workflow 51 All data we used is publicly and widely available on the Internet.
  52. 52. Rendering Pipeline Close-view Rendering 52
  53. 53. Rendering Pipeline Initial spherical geometries 53
  54. 54. Rendering Pipeline Depth correction 54
  55. 55. Rendering Pipeline Intersection removal 55
  56. 56. Rendering Pipeline Texturing individual geometry 56
  57. 57. Rendering Pipeline Texturing with alpha blending 57
  58. 58. Rendering Pipeline Rendering result in the fine detail 58
  59. 59. Rendering Pipeline Rendering result in the fine detail 59
  60. 60. Rendering Pipeline Rendering result in the fine detail 60
  61. 61. Rendering Pipeline Close-view Rendering 61
  62. 62. Rendering Pipeline Gap Alignment 62
  63. 63. Rendering Pipeline Gap Alignment 63
  64. 64. Rendering Pipeline Gap Alignment 64
  65. 65. Rendering Pipeline Seam Blending 65
  66. 66. Rendering Pipeline Seam Blending 66
  67. 67. Rendering Pipeline Seam Blending 67
  68. 68. Rendering Pipeline Street View vs. Satellite Images 68
  69. 69. Rendering Pipeline Street View vs. Satellite Images 69
  70. 70. Rendering Pipeline Street View vs. Satellite Images 70
  71. 71. Rendering Pipeline Gaussian Filtering 71
  72. 72. Rendering Pipeline Gaussian Filtering 72
  73. 73. Rendering Pipeline Gaussian Filtering 73
  74. 74. Rendering Pipeline Occlusion Test 74
  75. 75. Rendering Pipeline Occlusion Test 75
  76. 76. Rendering Pipeline Occlusion Test 76
  77. 77. LRU Cache Least Recently Used 77 5 adjacent street views are cached while users are walking. Each geometry has 131,074 vertices to be processed by the GPU
  78. 78. Limitations Inaccurate depth maps etc. 78
  79. 79. Deployment Geollery.com 79
  80. 80. Rendering Pipeline Experimental Features 80
  81. 81. Geollery/Social Street View has its own set of distinct offerings, as it is anchored within real-world settings, just mapped onto VR, whereas these are definitely more ‘fantasy’ type of arenas. In that way, as you have already done, I think there are multitude game challenges/tasks/feedback, like the balloons, to add in! Email feedback from pilot users 81
  82. 82. I think it’d be cool if you could see posts by people in real time, along with the establishment they’re in (like someone tweeting from inside McDonald’s or a movie theater), if that makes sense. Sort of like checking in to a place on Facebook Email feedback from pilot users 82
  83. 83. An interactive rendering pipeline of Fusing 360° Panoramas at two levels of detail. 83
  84. 84. Contributing a large-scale real-time system to Reconstruct a Mirrored World without the prior knowledge of any 3D models but only street view images and depth maps, which may be estimated from deep learning pipeline etc. 84
  85. 85. Establishing a web-based platform at Geollery.com for visualizing geotagged social media in a collaborative mixed-reality setting. 85
  86. 86. Discussion Use Case: Audio Tour 86 ...
  87. 87. Discussion Taking the Feedback: Geollery v2, Web3D & VR 2019 87 ...
  88. 88. Discussion Taking the Feedback 88
  89. 89. Challenge Global Market Restraint: Weak Content for XR 89
  90. 90. Research Goal Fuse the information from physical and virtual world 90 ACM Web3D '19
  91. 91. Future Directions The Ultimate XR Platform 91
  92. 92. Future Directions Fuses Past Events 92
  93. 93. Future Directions With the present 93
  94. 94. Future Directions And look into the future 94
  95. 95. Future Directions Change the way we communicate in 3D and consume the information 95
  96. 96. Future Directions Consume the information throughout the world 96
  97. 97. Acknowledgement NSF | MPower | UMIACS 97
  98. 98. Acknowledgement Coauthors 98
  99. 99. Thank you! Ruofei Du, David Li, and Amitabh Varshney {ruofei, dli7319, varshney}@cs.umd.edu | www.Geollery.com | CHI 2019 UMIACS THE AUGMENTARIUM VIRTUAL AND AUGMENTED REALITY LAB AT THE UNIVERSITY OF MARYLAND COMPUTER SCIENCE UNIVERSITY OF MARYLAND, COLLEGE PARK Thank you!
  100. 100. Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media Ruofei Du, David Li, and Amitabh Varshney {ruofei, dli7319, varshney}@umiacs.umd.edu | www.Geollery.com | CHI 2019 | Demo at D-2 (INT-40) UMIACS THE AUGMENTARIUM VIRTUAL AND AUGMENTED REALITY LAB AT THE UNIVERSITY OF MARYLAND COMPUTER SCIENCE UNIVERSITY OF MARYLAND, COLLEGE PARK Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media
  101. 101. User Study Background Interview (5 min) 101
  102. 102. User Study Quantitative Evaluation 102
  103. 103. User Study Quantitative Evaluation 103
  104. 104. User Study Post-interview 104
  105. 105. Post Interview Question 1/3 Suppose that we have a polished 3D social media platform like Geollery or Social Street View, would you like to use it? If so, how much time would you like to spend on it? 105
  106. 106. Post Interview Question 1/3 106
  107. 107. I would like to use it every day when I go to work, or travel during weekends. P6 / F 107
  108. 108. If it’s not distracting like Facebook and Instagram, I would use it every day on a couple of things. P17 / F 108
  109. 109. I am a follower on most social media sites. I would only join a 3D social media platform once my friends are there. P4 / M 109
  110. 110. If my friends are all on this, I can see myself spend a couple of hours every week. P12 / M 110
  111. 111. I don’t think I will use this. I prefer to use Yelp to see comments [of nearby restaurants] P12 / M 111
  112. 112. Post Interview Question 2/3 Can you imagine your use cases for Geollery and Social Street View? What would you like to use 3D social media platforms for? 112
  113. 113. I would like to use it for the food in different restaurants. I am always hesitating of different restaurants. It will be very easy to see all restaurants with street views. In Yelp, I can only see one restaurant at a time. P6 / F 113
  114. 114. [I will use it for] exploring new places. If I am going on vacation somewhere, I could immerse myself into the location. If there are avatars around that area, I could ask questions. P1 / M 114
  115. 115. I think it (Geollery) will be useful for families. I just taught my grandpa how to use Facetime last week and it would great if I could teleport to their house and meet with them, then we could chat and share photos with our avatars. P2 / F 115
  116. 116. … for communicating with my families, maybe, and distant friends, [so] they can see New York. And, getting to know more people, connecting with people based on similar interests. P2 / F 116
  117. 117. Post Interview Question 3/3 If you were a designer or product manager for Geollery or Social Street View, what features would you like to add to the systems? 117
  118. 118. A mapping of the texture, high-resolution texture, will be great. P12 / M 118
  119. 119. if there is a way to unify the interaction between them, there will be more realistic buildings [and] you could have more roof structures. Terrains will be interesting to add on. P18 / M 119
  120. 120. I would like to see kitties and puppies running around, and birds flying in the air P13 / F 120
  121. 121. I could also add a bike, add a vehicle, a motorcycle in Geollery, this will add some fun. P17 / F 121

×