Proceedings (Full paper reviewed) Tomohiro Fukuda, Hideki Nada, Haruo Adachi, Shunta Shimizu, Chikako Takei, Yusuke Sato, Nobuyoshi Yabuki, and Ali Motamedi: 2017, Integration of a Structure from Motion into Virtual and Augmented Reality for Architectural and Urban Simulation: Demonstrated in Real Architectural and Urban Projects, Future Trajectories of Computation in Design: 17th International Conference CAAD Futures 2017, p.596, 2017.7 Book (Book contribution) Tomohiro Fukuda, Hideki Nada, Haruo Adachi, Shunta Shimizu, Chikako Takei, Yusuke Sato, Nobuyoshi Yabuki, and Ali Motamedi: 2017, Integration of a Structure from Motion into Virtual and Augmented Reality for Architectural and Urban Simulation: Demonstrated in Real Architectural and Urban Projects, Computer-Aided Architectural Design - Future Trajectories，pp.60-77，Springer (Communications in Computer and Information Science 724), ISSN 1865-0929，ISBN 978-981-10-5196-8，2017.7 Computational visual simulations are extremely useful and powerful tools for decision-making. The use of virtual and augmented reality (VR/AR) has become a common phenomenon due to real-time and interactive visual simulation tools in architectural and urban design studies and presentations. In this study, a demonstration is performed to integrate structure from motion (SfM) into VR and AR. A 3D modeling method is explored by SfM under real-time rendering as a solution for the modeling cost in large-scale VR. The study examines the application of camera parameters of SfM to realize an appropriate registration and tracking accuracy in marker-less AR to visualize full-scale design projects on a planned construction site. The proposed approach is applied to plural real architectural and urban design projects, and results indicate the feasibility and effectiveness of the proposed approach.

1. Integration of a Structure from Motion
into Virtual and Augmented Reality
for Architectural and Urban Simulation
July 13 2017
Tomohiro Fukuda1 Hideki Nada2 Haruo Adachi2
Shunta Shimizu3 Chikako Takei3 Yusuke Sato1
Nobuyoshi Yabuki1 Ali Motamedi1
1
Demonstrated in Real Architectural and Urban Projects
1 Osaka University, Osaka, Japan
2 Sakaiminato City Office,Tottori, Japan
3 Forum8 Co., Ltd.,Tokyo, Japan

2. Contents
1. Introduction
2. SfM-based Modeling for VR
3. Using Camera Parameters of SfM for Marker-less AR
4. Conclusions
2

3. Virtual and Augmented Reality
 Constructs 3D past and future virtual spaces with real-time and
interactive operation.
 Considerable amount of time and expense is entailed on an
urban scale to construct and integrate 3D objects for all virtual
spaces.
3
Real
Env.
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Env. = VR
Mixed Reality (MR)
Simplified representation of a "virtuality continuum“ by Milgram and Kishino (1994)
VR Azuchi Castle prj.
CAAD Futures 2015, Sao Paulo
Housing Design prj.
eCAADe 2015, Vienna

4. Virtual and Augmented Reality
 Overlays 3D virtual objects on real world images
 Can help visualize full-scale design projects on a planning site
 Must track the geometric registration between live video and 3D
objects to accurately render the outdoor AR
4
Real
Env.
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Env. = VR
Mixed Reality (MR)
Simplified representation of a "virtuality continuum“ by Milgram and Kishino (1994)
Sensor-based AR
eCAADe2006, Volos
Marker-based AR
2015 Int’l WS on Computing in Civil Engineering, Texas
17 22 27℃

5. SfM (Structure from Motion)
 Estimates 3D structures from 2D sequential images that are
coupled to local motion signals
 Requires only familiar equipment (Smart phones), free or low-cost
software
5
Google Maps
Target Building
Shooting routes and directions
Camera parameter
Point clouds
点群
Dense point clouds
1,335,958 points Camera Parameter
3,194,585 faces
1,598568 vertices

6. Objective
Integration of SfM into VR and AR:
 A 3D modeling method by SfM are explored for real-time
rendering to realize less modeling cost in large-scale VR
 Reduce a large number of generated point clouds and meshes
for the frame rate of real-time rendering
 Camera parameters of SfM are explored for a marker-less AR
to realize registration and tracking accuracy to visualize full-
scaled architectural design projects
 Apply SfM-based AR to real architectural design projects to
examine the uses and limitations
6

7. Contents
1. Introduction
2. SfM-based Modeling for VR
3. Using Camera Parameters of SfM for Marker-less AR
4. Conclusions
7

8. Urban Design Project:
Mizuki Shigeru Road
Renewal
4

13. Challenges
 Aging
 Barrier-free

14. Schematic renovation plan
Creating a hospitable and entertaining road
that everyone would like to visit
Shigeru Mizuki MuseumJR Sakaiminato st.
800ｍ
水木マンガの世界
森にすむ妖怪たち
神仏･吉凶を司る妖怪たち 身近なところにひそむ妖怪たち
川や水辺 野や里 山 海辺 町
家に棲む
妖怪たち
 Creation of a one-way and S-shaped roadway
 Creation of a wider sidewalk space
 Re-accommodation of the bronze statues within the new
zone concept
 Improvement of cityscape with a sense of unity
 Night landscape design

15. 15

16. 16

17. VR production
Challenges
 3D digital modeling of 153 bronze statues
 Possess different shapes and included complicated shapes with holes
 Conventional method
 Modeling using 3DCG software demands enormous time and cost
resources
17
 Construction design study and public meeting to build
consensus
Representation of a Human’s Motion and Direction by 2D images in VR
(Fukuda et al., Int’l J. of Arch. Comput. 7(2), 2009)

18. Photography for SfM
Conditions
 Shoot on cloudy days
 To generate a shadow-less texture at 3D
modeling by SfM to ensure optical consistency
between VR and 3D objects by SfM
 Shoot at times of less tourists
 Not to shoot noise elements
Role sharing
 Sakaiminato city office staffs took 10,000
photos of 153 statues in a month
 VR experts worked away from the project site
18

19. Photographs for SfM
19

20. SfM-based Statues Modeling
20
High
4286 polygons
Medium
1025 polygons
Low
298 polygons
 Software: PhotoScan, Remake,
UC-win/Road SfM plug-in
 Shooting time: 10-20 min./statue
 Production time: Automatically generated 30 min.
Editing 30min.

21. SfM-based Statues Modeling
 153 bronze statues are ranked
into 3 categories:
 High 27 Med 46 Low 80
21
OS Windows7 Professional 64-bit
CPU Intel Corei7-3770 3.40GHz
RAM 16GB
GPU NVIDIA GeForce GTX 670 VRAM 4GB)
Res. 1920x1080
100 statues
Accuracy fps
High 16.4
Med. 24.1
Low 27.1
 When all the high-accuracy statues are
installed, realizing real-time rendering is
difficult.

22. Results: 3D model by SfM
22
河童
ねずみ男 ガラッパ 小豆洗い
鬼太郎と
目玉おやじ
かみきりと
一反木綿
袖引小僧龍水木しげる先生 執筆中
(N=27) (N=46) (N=80)

23. SfM-based Roadside Building Modeling
23
Conventional method
Polygons 19
Vertices 48
 120 buildings along the roadside▶ Difficult to realize real-time
rendering using SfM-based 3D model
SfM (e.g. PhotoScan)
Polygons 2.4 million
Vertices 1.2 million

24. SfM-based Roadside Building Modeling
24
Conventional method
 Editing work still involves a considerable amount of time.
 The roadside buildings were finally created by using the
conventional method.
SfM-Photoscan SfM-Remake
Lack of content: Elements to be modeled are not
modeled by SfM (roof)
Inadequate content: Geometry is not correct
although it is modeled by SfM (signs, arcade)
Surplus content: Unnecessary objects are
modeled (wire contains sky elements)

25. 25
3D model
Number of polygons 965,472
Number of vertices 929,325
Texture
Number of textures 885
Number of texture pixels 65,530,368
Data amount 1.5 GB
VRAM use when running VR application 667.4 MB
Total
Data amount 1.35 GB
Frame rate 30-50 fps

26. 26

27. 27

28. 28水木しげるロードリニューアルイメージ映像

29. Contents
1. Introduction
2. SfM-based Modeling for VR
3. Using Camera Parameters of SfM for Marker-less AR
4. Conclusions
29

30. Marker-less AR: System Flow
30

31. Marker-less AR: System Flow
31
Photographs from viewpoints

32. Marker-less AR: System Flow
32
Store position and orientation data of all camera
3D point cloud reconstructed by SfM

33. Marker-less AR: System Flow
33
Define relative coordinates of 3D
design objects using SfM model

34. Marker-less AR: System Flow
34
Start video capture (OpenCV 2.4)

35. Marker-less AR: System Flow
35
 Extract features of the live video image
 Compare them with features in DB
using SURF (Speeded-up Robust Features)
 Extract the camera’s position and
orientation of the most similar image

36. Marker-less AR: System Flow
36
Render 3D virtual objects precisely in AR
using referred position and orientation data

37. Marker-less AR: System Flow
37
 Motion vectors for tracking are calculated by the optical flow technique.
 Positions of points on the screen and the corresponding world coordinates of those
points are calculated in real time.

38. Application in Building Design prj.
38
 To study the facade and approach of the new office building
at the preliminary design stage (Jan – Mar 2016)
 Demolish the existing office building and build a new three-
story office building in Osaka, Japan
0 20m

39. Interview survey
Advantage
 The size of the building could be intuitively comprehended at
the construction site
 It was possible to obtain deeper communication in the
preliminary design stage.
Disadvantage
 Difficulty of viewing the exterior as well as the building model
through the tablet LCD display.
 Walking freely by using a video see-through type HMD is difficult
because the surrounding environment is not adequately visible.
 Using an optical see-through HMD, a problem exists wherein it is
difficult to view the AR due to the effect of sunlight.
39

40. Interview survey
 Although the size of the rendered 3D model is accurate, an
opinion persisted that this was not possible.
 The validity of the size of the designed building can be
explained by comparing it with the size of the adjacent
building in the experiment.
40

41. Contents
1. Introduction
2. SfM-based Modeling for VR
3. Using Camera Parameters of SfM for Marker-less AR
4. Conclusions
41

42. Conclusions
 Integrating SfM into VR and AR is demonstrated for
architectural and urban visual simulations.
 A modeling method using SfM is developed to model a number
of 3D objects under real-time rendering conditions. The total
data amount of bronze statues is suppressed by setting three
reproduction ranks. A large-scale VR application can be
constructed with decreasing the modeling cost.
 An AR system using SfM camera parameters for registration is
developed to visualize design projects. Design stakeholders
could study the size and facade of the office building by using
the full-scale displayed AR at the planning site. It is possible to
obtain a more constructive communication among
stakeholders.
 A future study should be examined further integration into VR
and AR based on the technology to restore a 3D model in real
time from camera images (e.g., Simultaneous Localization and Mapping).
42

43. Acknowledgements
 The research was partly supported by JSPS KAKENHI Grant
Number JP25350010 and JP26-04368, and joint research
funding between Sakaiminato city office and Osaka University.
 We would also like to take the opportunity to thank Kyokuyo
Electric Co., Ltd. and atelier DoN to apply our proposed AR
system to the building design project.
 We thank all the participants for their generous assistance in
conducting projects and experiments.
4343Mizuki Shigeru Road (Jul 2017) Kyokuyo Electric Co. Ltd. New head office (Jul 2017)