Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Integration of a Structure from Motion into Virtual and Augmented Reality for Architectural and Urban Simulation

3,058 views

Published on

Proceedings (Full paper reviewed)
Tomohiro Fukuda, Hideki Nada, Haruo Adachi, Shunta Shimizu, Chikako Takei, Yusuke Sato, Nobuyoshi Yabuki, and Ali Motamedi: 2017, Integration of a Structure from Motion into Virtual and Augmented Reality for Architectural and Urban Simulation: Demonstrated in Real Architectural and Urban Projects, Future Trajectories of Computation in Design: 17th International Conference CAAD Futures 2017, p.596, 2017.7

Book (Book contribution)
Tomohiro Fukuda, Hideki Nada, Haruo Adachi, Shunta Shimizu, Chikako Takei, Yusuke Sato, Nobuyoshi Yabuki, and Ali Motamedi: 2017, Integration of a Structure from Motion into Virtual and Augmented Reality for Architectural and Urban Simulation: Demonstrated in Real Architectural and Urban Projects, Computer-Aided Architectural Design - Future Trajectories,pp.60-77,Springer (Communications in Computer and Information Science 724), ISSN 1865-0929,ISBN 978-981-10-5196-8,2017.7

Computational visual simulations are extremely useful and powerful tools for decision-making. The use of virtual and augmented reality (VR/AR) has become a common phenomenon due to real-time and interactive visual simulation tools in architectural and urban design studies and presentations. In this study, a demonstration is performed to integrate structure from motion (SfM) into VR and AR. A 3D modeling method is explored by SfM under real-time rendering as a solution for the modeling cost in large-scale VR. The study examines the application of camera parameters of SfM to realize an appropriate registration and tracking accuracy in marker-less AR to visualize full-scale design projects on a planned construction site. The proposed approach is applied to plural real architectural and urban design projects, and results indicate the feasibility and effectiveness of the proposed approach.

  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Integration of a Structure from Motion into Virtual and Augmented Reality for Architectural and Urban Simulation

  1. 1. Integration of a Structure from Motion into Virtual and Augmented Reality for Architectural and Urban Simulation July 13 2017 Tomohiro Fukuda1 Hideki Nada2 Haruo Adachi2 Shunta Shimizu3 Chikako Takei3 Yusuke Sato1 Nobuyoshi Yabuki1 Ali Motamedi1 1 Demonstrated in Real Architectural and Urban Projects 1 Osaka University, Osaka, Japan 2 Sakaiminato City Office,Tottori, Japan 3 Forum8 Co., Ltd.,Tokyo, Japan
  2. 2. Contents 1. Introduction 2. SfM-based Modeling for VR 3. Using Camera Parameters of SfM for Marker-less AR 4. Conclusions 2
  3. 3. Virtual and Augmented Reality  Constructs 3D past and future virtual spaces with real-time and interactive operation.  Considerable amount of time and expense is entailed on an urban scale to construct and integrate 3D objects for all virtual spaces. 3 Real Env. Augmented Reality (AR) Augmented Virtuality (AV) Virtual Env. = VR Mixed Reality (MR) Simplified representation of a "virtuality continuum“ by Milgram and Kishino (1994) VR Azuchi Castle prj. CAAD Futures 2015, Sao Paulo Housing Design prj. eCAADe 2015, Vienna
  4. 4. Virtual and Augmented Reality  Overlays 3D virtual objects on real world images  Can help visualize full-scale design projects on a planning site  Must track the geometric registration between live video and 3D objects to accurately render the outdoor AR 4 Real Env. Augmented Reality (AR) Augmented Virtuality (AV) Virtual Env. = VR Mixed Reality (MR) Simplified representation of a "virtuality continuum“ by Milgram and Kishino (1994) Sensor-based AR eCAADe2006, Volos Marker-based AR 2015 Int’l WS on Computing in Civil Engineering, Texas 17 22 27℃
  5. 5. SfM (Structure from Motion)  Estimates 3D structures from 2D sequential images that are coupled to local motion signals  Requires only familiar equipment (Smart phones), free or low-cost software 5 Google Maps Target Building Shooting routes and directions Camera parameter Point clouds 点群 Dense point clouds 1,335,958 points Camera Parameter 3,194,585 faces 1,598568 vertices
  6. 6. Objective Integration of SfM into VR and AR:  A 3D modeling method by SfM are explored for real-time rendering to realize less modeling cost in large-scale VR  Reduce a large number of generated point clouds and meshes for the frame rate of real-time rendering  Camera parameters of SfM are explored for a marker-less AR to realize registration and tracking accuracy to visualize full- scaled architectural design projects  Apply SfM-based AR to real architectural design projects to examine the uses and limitations 6
  7. 7. Contents 1. Introduction 2. SfM-based Modeling for VR 3. Using Camera Parameters of SfM for Marker-less AR 4. Conclusions 7
  8. 8. Urban Design Project: Mizuki Shigeru Road Renewal 4
  9. 9. Challenges  Aging  Barrier-free
  10. 10. Schematic renovation plan Creating a hospitable and entertaining road that everyone would like to visit Shigeru Mizuki MuseumJR Sakaiminato st. 800m 水木マンガの世界 森にすむ妖怪たち 神仏・吉凶を司る妖怪たち 身近なところにひそむ妖怪たち 川や水辺 野や里 山 海辺 町 家に棲む 妖怪たち  Creation of a one-way and S-shaped roadway  Creation of a wider sidewalk space  Re-accommodation of the bronze statues within the new zone concept  Improvement of cityscape with a sense of unity  Night landscape design
  11. 11. 15
  12. 12. 16
  13. 13. VR production Challenges  3D digital modeling of 153 bronze statues  Possess different shapes and included complicated shapes with holes  Conventional method  Modeling using 3DCG software demands enormous time and cost resources 17  Construction design study and public meeting to build consensus Representation of a Human’s Motion and Direction by 2D images in VR (Fukuda et al., Int’l J. of Arch. Comput. 7(2), 2009)
  14. 14. Photography for SfM Conditions  Shoot on cloudy days  To generate a shadow-less texture at 3D modeling by SfM to ensure optical consistency between VR and 3D objects by SfM  Shoot at times of less tourists  Not to shoot noise elements Role sharing  Sakaiminato city office staffs took 10,000 photos of 153 statues in a month  VR experts worked away from the project site 18
  15. 15. Photographs for SfM 19
  16. 16. SfM-based Statues Modeling 20 High 4286 polygons Medium 1025 polygons Low 298 polygons  Software: PhotoScan, Remake, UC-win/Road SfM plug-in  Shooting time: 10-20 min./statue  Production time: Automatically generated 30 min. Editing 30min.
  17. 17. SfM-based Statues Modeling  153 bronze statues are ranked into 3 categories:  High 27 Med 46 Low 80 21 OS Windows7 Professional 64-bit CPU Intel Corei7-3770 3.40GHz RAM 16GB GPU NVIDIA GeForce GTX 670 VRAM 4GB) Res. 1920x1080 100 statues Accuracy fps High 16.4 Med. 24.1 Low 27.1  When all the high-accuracy statues are installed, realizing real-time rendering is difficult.
  18. 18. Results: 3D model by SfM 22 河童 ねずみ男 ガラッパ 小豆洗い 鬼太郎と 目玉おやじ かみきりと 一反木綿 袖引小僧龍水木しげる先生 執筆中 (N=27) (N=46) (N=80)
  19. 19. SfM-based Roadside Building Modeling 23 Conventional method Polygons 19 Vertices 48  120 buildings along the roadside▶ Difficult to realize real-time rendering using SfM-based 3D model SfM (e.g. PhotoScan) Polygons 2.4 million Vertices 1.2 million
  20. 20. SfM-based Roadside Building Modeling 24 Conventional method  Editing work still involves a considerable amount of time.  The roadside buildings were finally created by using the conventional method. SfM-Photoscan SfM-Remake Lack of content: Elements to be modeled are not modeled by SfM (roof) Inadequate content: Geometry is not correct although it is modeled by SfM (signs, arcade) Surplus content: Unnecessary objects are modeled (wire contains sky elements)
  21. 21. 25 3D model Number of polygons 965,472 Number of vertices 929,325 Texture Number of textures 885 Number of texture pixels 65,530,368 Data amount 1.5 GB VRAM use when running VR application 667.4 MB Total Data amount 1.35 GB Frame rate 30-50 fps
  22. 22. 26
  23. 23. 27
  24. 24. 28水木しげるロードリニューアルイメージ映像
  25. 25. Contents 1. Introduction 2. SfM-based Modeling for VR 3. Using Camera Parameters of SfM for Marker-less AR 4. Conclusions 29
  26. 26. Marker-less AR: System Flow 30
  27. 27. Marker-less AR: System Flow 31 Photographs from viewpoints
  28. 28. Marker-less AR: System Flow 32 Store position and orientation data of all camera 3D point cloud reconstructed by SfM
  29. 29. Marker-less AR: System Flow 33 Define relative coordinates of 3D design objects using SfM model
  30. 30. Marker-less AR: System Flow 34 Start video capture (OpenCV 2.4)
  31. 31. Marker-less AR: System Flow 35  Extract features of the live video image  Compare them with features in DB using SURF (Speeded-up Robust Features)  Extract the camera’s position and orientation of the most similar image
  32. 32. Marker-less AR: System Flow 36 Render 3D virtual objects precisely in AR using referred position and orientation data
  33. 33. Marker-less AR: System Flow 37  Motion vectors for tracking are calculated by the optical flow technique.  Positions of points on the screen and the corresponding world coordinates of those points are calculated in real time.
  34. 34. Application in Building Design prj. 38  To study the facade and approach of the new office building at the preliminary design stage (Jan – Mar 2016)  Demolish the existing office building and build a new three- story office building in Osaka, Japan 0 20m
  35. 35. Interview survey Advantage  The size of the building could be intuitively comprehended at the construction site  It was possible to obtain deeper communication in the preliminary design stage. Disadvantage  Difficulty of viewing the exterior as well as the building model through the tablet LCD display.  Walking freely by using a video see-through type HMD is difficult because the surrounding environment is not adequately visible.  Using an optical see-through HMD, a problem exists wherein it is difficult to view the AR due to the effect of sunlight. 39
  36. 36. Interview survey  Although the size of the rendered 3D model is accurate, an opinion persisted that this was not possible.  The validity of the size of the designed building can be explained by comparing it with the size of the adjacent building in the experiment. 40
  37. 37. Contents 1. Introduction 2. SfM-based Modeling for VR 3. Using Camera Parameters of SfM for Marker-less AR 4. Conclusions 41
  38. 38. Conclusions  Integrating SfM into VR and AR is demonstrated for architectural and urban visual simulations.  A modeling method using SfM is developed to model a number of 3D objects under real-time rendering conditions. The total data amount of bronze statues is suppressed by setting three reproduction ranks. A large-scale VR application can be constructed with decreasing the modeling cost.  An AR system using SfM camera parameters for registration is developed to visualize design projects. Design stakeholders could study the size and facade of the office building by using the full-scale displayed AR at the planning site. It is possible to obtain a more constructive communication among stakeholders.  A future study should be examined further integration into VR and AR based on the technology to restore a 3D model in real time from camera images (e.g., Simultaneous Localization and Mapping). 42
  39. 39. Acknowledgements  The research was partly supported by JSPS KAKENHI Grant Number JP25350010 and JP26-04368, and joint research funding between Sakaiminato city office and Osaka University.  We would also like to take the opportunity to thank Kyokuyo Electric Co., Ltd. and atelier DoN to apply our proposed AR system to the building design project.  We thank all the participants for their generous assistance in conducting projects and experiments. 4343Mizuki Shigeru Road (Jul 2017) Kyokuyo Electric Co. Ltd. New head office (Jul 2017)

×