Keynote talk about AR in and around Buildings held at the "Marketingtag der Fertigbauindustrie" the annual gathering of the German prefabricated building industry @Fertighaus

1. © Fraunhofer Institute for Applied Information Technology FIT
Dr. Leif Oppermann, Head of Mixed and Augmented Reality Solutions
Seite 1
MIXED-, VIRTUAL- UND AUGMENTED REALITY
GETTING OUTSIDE AND BACK INTO BUILDINGS

2. © Fraunhofer Institute for Applied Information Technology FIT
Fraunhofer FIT
 Fraunhofer is the largest organization for applied research in
Europe
 72 institutes, 25.000 employees
 Invented MP3 Audio Compression
 Fraunhofer Institute for Applied Information Technology FIT
 Human-centered computing
 Cooperation systems
 Located on Institute Center Birlinghoven
 Since 1968 one of the largest German centers
in IT and applied mathematics
 50 years of human-centered digitalisation
 20+ years of AR / MR / XR / WhatevR
Seite 2
Old building needs renovation

3. © Fraunhofer Institute for Applied Information Technology FIT
Enabling technologies

4. © Fraunhofer Institute for Applied Information Technology FIT
Not entirely new, but still…
Seite 4
Ivan Sutherland,
The Ultimate Display / Sword of Damocles, 1968

5. © Fraunhofer Institute for Applied Information Technology FIT
Graphics Performance 1998 vs. 2008, Trend to Mobile
Seite 5
Source: Mark Billinghurst

6. © Fraunhofer Institute for Applied Information Technology FIT
Perspective of our Research Group
(Mixed and Augmented Reality Solutions)
 Ever improving mobile technology
 Enabling Virtual and Augmented Reality
 What can you actually do with it?
 Approaching the problem through building
and evaluating representative prototypes
 Interweaving the Real and the Digital
 Increasingly mobile and location-based
 Augmented Reality (AR) is a variation
of Virtual Reality (VR)
 VR replaces perception
 AR extends/augments it
 MR embraces both
Seite 6
The Lawnmower Man, Columbia (1992)
TimeWarp, Fraunhofer FIT (2010)

7. © Fraunhofer Institute for Applied Information Technology FIT
Getting AR outside in the 2000s
Seite 7
 From enhanced rooms
 To buildings
 To the campus
 Into the city
 From marker to GPS & compass
Architecture (ARTHUR)
https://www.youtube.com/watch?v=v4sK7q70dT8
Building Maintenance (CoSpaces)
https://www.youtube.com/watch?v=AJBiGqdIUts
Pervasive Gaming, Crossmedia Gaming, mobile AR (IPerG)
https://www.youtube.com/watch?v=CWhZhB2crH8 , https://www.fit.fraunhofer.de/de/fb/cscw/projects/iperg.html
Mobile AR in the City (IPCity)
https://www.fit.fraunhofer.de/de/fb/cscw/projects/ipcity.html
https://www.youtube.com/watch?v=WjjHMqSGPpE

8. © Fraunhofer Institute for Applied Information Technology FIT
Seite 8
Visualising Building Models (Auto AR) Models courtesy of FIDUCIA
Presented at:
Augmented World Expo
Europe 2017

9. © Fraunhofer Institute for Applied Information Technology FIT
Auto AR by Mixed and Augmented Reality Solutions @Fraunhofer_FIT
Seite 9
YouTube: https://www.youtube.com/watch?v=M-XQJCCq5lg

10. © Fraunhofer Institute for Applied Information Technology FIT
Motivation for Auto AR
 Group started at the verge of CSCW and AR
 First projects in construction (indoors)
 Innovative use of interaction devices
 Increasingly mobile and location-based
 10+ years in mobile mixed reality projects
 Idea: Apply that experience to construction
 Initial cooperation with Bauhaus
 Joint projects with HIT Lab NZ
 Received Fraunhofer Strategic Invest
 First presented at BAU 2015
 More examples today
Seite 10

11. © Fraunhofer Institute for Applied Information Technology FIT
Construction planning now and then
 Problem: reading blue-prints is a specialist skill
 Two typical tools for giving a first impression of the planned space:
 Miniature-models
 Virtual renderings (still or video)
 Cons: building not seen in surrounding environment, expensive,
little flexibility to react to changes, outdates quickly
Seite 11

12. © Fraunhofer Institute for Applied Information Technology FIT
Enters Building Information Modeling (BIM)
 BIM is a method, not a software! (Wernicke, 2011)
 New way for cooperation, based on a common understanding
(in planning, building, maintaining)
 Sustainable process
(Time, cost & quality)
 Currently optional,
but soon mandatory
 Building data is standardised
and easily available
Seite 12

13. © Fraunhofer Institute for Applied Information Technology FIT
Auto AR System Overview
 Augmented Reality for construction projects
 Integrate into Building Information Modeling (BIM) processes,
i.e. use existing data, and connect to engineering workflows
 Uses
 360° panoramic camera images
 PC and commercial game-engine for interactive realtime 3D
 Various displays to show augmented view, e.g. Oculus Rift
 Interface to engineers hardware and software
 High precision surveying equipment
 CAD, etc.
 It is mobile on a car -> AutoAR
Seite 13

14. © Fraunhofer Institute for Applied Information Technology FIT
Preparations start from existing data
here: CAD / Revit
Seite 14
Low-Poly FBX
Convert

15. © Fraunhofer Institute for Applied Information Technology FIT
Site Survey
Seite 15
 Drive around site as desired
 Record camera and sensor data
 Live-Preview
 Experience on-site
 Useful for some clients
 But mostly for our team
SITE

16. © Fraunhofer Institute for Applied Information Technology FIT
Replay in the office
Seite 16
 Off-site visualisation
 Flexible view:
 video and sensor playback
 free head movement
 Multiple visualisations in parallel
(monitor, HMD, beamer)
 Compare model variations
 Easily change models
 Capture for video editing
 Process for 360° versions
 All this with data that you already have!

17. © Fraunhofer Institute for Applied Information Technology FIT
Auto AR Example Results
Seite 17
Visualisation made for
More details:
BIMiD-Talk

18. © Fraunhofer Institute for Applied Information Technology FIT
Auto AR Example Results
Own campus at Schloss Birlinghoven
Seite 18
YouTube: https://www.youtube.com/watch?v=YYhplphGZOc

19. © Fraunhofer Institute for Applied Information Technology FIT
Auto AR Wrap-up
 3D data from Building Information Modeling is well suited
 Our core work with Auto AR usually just takes days – weeks
 Auto AR supports understanding of planned building on site
 Complex tear-down and rebuild of existing campus
 Fassade in context of surrounding buildings
 Proximity to other buildings and their uses
 So if you have 3D data of your buildings & structures
 … and would like to have an attractive
AutoAR video or 360° experience
 Please get in touch
Seite 19
https://www.presseportal.de/pm/60247/3340076

20. © Fraunhofer Institute for Applied Information Technology FIT
Another year, another use-case
Seite 20
Presented at:
Augmented World Expo
Europe 2018

21. © Fraunhofer Institute for Applied Information Technology FIT
Getting back inside with 3D building models
Seite 21
Models courtesy of IZB.LD

22. © Fraunhofer Institute for Applied Information Technology FIT
Unique opportunity : planned construction
 Complete set of building models
 Rooms
 Pipes and supply engineering
 Various studies and theses
 Different applications
 Different devices
 Technology tests
 Indoor tracking
 User acceptance
Seite 22

23. © Fraunhofer Institute for Applied Information Technology FIT
During construction
Seite 23
HUB conference, Berlin 2016

24. © Fraunhofer Institute for Applied Information Technology FIT
Building Information Modeling to Go
Seite 24
YouTube: https://youtu.be/Yxq14NrCShU

25. © Fraunhofer Institute for Applied Information Technology FIT
 Experiencing interior design and 3D models
 Spatial impression even during construction
 Done in cooperation with Vario
Adding furniture
Seite 25

26. © Fraunhofer Institute for Applied Information Technology FIT
AR Furniture
Seite 26
YouTube: https://youtu.be/XjrGzuoQ8jQ

27. © Fraunhofer Institute for Applied Information Technology FIT
Study of collaborative indoor AR
 In a realisitic scenario
 Real building
 Real data
 Collaborative tasks
 1.) Building maintenance
 2.) Interior design / furniture
 Making decisions, discussing placements, etc.
 Hand-held (Tango tablet) vs. hands-free (HoloLens)
 Which device is preferred for which task?
Seite 27
Full text: https://www.mdpi.com/2414-4088/3/2/23

28. © Fraunhofer Institute for Applied Information Technology FIT
Setup
 16 participants in groups of 2
 Architects, researchers, student assistants
 14 male, 2 female
 10 unfamiliar with the devices
 Four tasks
 Two on tablet
 Two on smart glasses
 Questionnaires and interviews
 Live-observer and video-observer
 About 30-45 per test-run
Seite 28

29. © Fraunhofer Institute for Applied Information Technology FIT
Results 1/2
Seite 29

30. © Fraunhofer Institute for Applied Information Technology FIT
Results 2/2
 Tango Tablet clearly favoured in our study
 More familiar interaction, handy device
 Eases face to face discussion
 Sadly, Tango has been discontinued…
 … and ARCore doesn‘t support Area Learning (yet?)
 Findings for remote collaboration would likely differ
Seite 30

31. © Fraunhofer Institute for Applied Information Technology FIT
… to be continued
 Leif Oppermann
 Fraunhofer FIT
Schloss Birlinghoven
53754 Sankt Augustin
 http://www.fit.fraunhofer.de/mars
Seite 31
c.f. recent study about VR/AR for the German Bundestag
https://twitter.com/TABundestag/status/1143542955707445248