Augmented Reality Prof. Charles Woodward VTT Technical Research Centre of Finland

Outline Introduction VTT Virtual and Augmented Reality Applications at VTT Entertainment, marketing Interior design, architecture Mobile Augmented Reality Production applications Augmented Reality Future Some new ideas Enabling technologies Trends and Drivers

Introduction

VTT

Virtual and Augmented Reality

Applications at VTT

Entertainment, marketing

Interior design, architecture

Mobile Augmented Reality

Production applications

Augmented Reality Future

Some new ideas

Enabling technologies

Trends and Drivers

INTRODUCTION

VTT Profile 7 Knowledge Clusters Digital Information Systems Telecommunications Microtechnologies and Sensors Materials and Building Industrial Systems Biotechnology Energy and Pulp&Paper Figures 2 700 employees 233 M€ research budget 45 Knowledge Centres Basic research Applied research Development work VTT INDUSTRY UNIVERSITIES

7 Knowledge Clusters

Digital Information Systems

Telecommunications

Microtechnologies and Sensors

Materials and Building

Industrial Systems

Biotechnology

Energy and Pulp&Paper

Figures

2 700 employees

233 M€ research budget

45 Knowledge Centres

Virtual and Augmented Reality at VTT VR/AR Roadmap report 2005 Fourteen VTT research teams contributing Applications from entertainment to digital media, construction to production, ships, military, etc. Technology, incl. Core VR/AR solutions Machine vision, feature tracking Multimodality, 3D audio etc. Motion tracking, avatars, sensors GSM/GPS positioning, GIS systems Ambient intelligence, context sensitivity, … Special strenghts Multi-disciplinary research Strong industry relations and experience

VR/AR Roadmap report 2005

Fourteen VTT research teams contributing

Applications from entertainment to digital media, construction to production, ships, military, etc.

Technology, incl.

Core VR/AR solutions

Machine vision, feature tracking

Multimodality, 3D audio etc.

Motion tracking, avatars, sensors

GSM/GPS positioning, GIS systems

Ambient intelligence, context sensitivity, …

Special strenghts

Multi-disciplinary research

Strong industry relations and experience

Virtual Reality (Introduction) Devices, incl. Head Mounted Displays (HMD), data gloves, motion trackers, ... CAVE systems “ Lumeportti” - VTT’s proprietary CAVE for 3D data visualisation Utilises video projectors, networked PCs and low-cost VR technology Provides steroscopic walk-through functionality with gesture-based interaction Applications e.g. space ship simulation for Alenia Spazio

Devices, incl.

Head Mounted Displays (HMD), data gloves, motion trackers, ...

CAVE systems

“ Lumeportti” - VTT’s proprietary CAVE for 3D data visualisation

Utilises video projectors, networked PCs and low-cost VR technology

Provides steroscopic walk-through functionality with gesture-based interaction

Applications e.g. space ship simulation for Alenia Spazio

Augmented Reality Augmented Reality (AR) Mixes virtual objects with user’s view of real world Properties: real-time, interactive, 3D Related technologies Augmented Virtuality Mediated Reality Diminished Reality Mixed Reality Display devices PC screen + web camera (”Magic mirror”) Data glasses (video or optical see through HMD) Hand held devices, camera phones (”Magic lense”) Reality Augmented Reality Augmented Virtuality Virtual Reality Mixed Reality

Augmented Reality (AR)

Mixes virtual objects with user’s view of real world

Properties: real-time, interactive, 3D

Related technologies

Augmented Virtuality

Mediated Reality

Diminished Reality

Mixed Reality

Display devices

PC screen + web camera (”Magic mirror”)

Data glasses (video or optical see through HMD)

Hand held devices, camera phones (”Magic lense”)

AUGMENTED REALITY – APPLICATIONS AT VTT

ARCatalogue Enhanced ”Magic Book” application In real marketing use by e.g. Lappset Group ARCipher Implementation on camera phones Augmenting on matrix codes AR in Marketing / Media

ARCatalogue

Enhanced ”Magic Book” application

In real marketing use by e.g. Lappset Group

ARCipher

Implementation on camera phones

Augmenting on matrix codes

CamBall Augmented networked table tennis SymBall Virtual table tennis for camera phones StereoGames 3D Stereo display enhancement AR in Games / Entertainment

CamBall

Augmented networked table tennis

SymBall

Virtual table tennis for camera phones

StereoGames

3D Stereo display enhancement

AR at Homes / Interiors Basic solution Virtual furniture object tied to each marker ARInteriors Augmented digital images ARPhone Virtual furniture on phone’s camera image Downloading models by matrix codes

Basic solution

Virtual furniture object tied to each marker

ARInteriors

Augmented digital images

ARPhone

Virtual furniture on phone’s camera image

Downloading models by matrix codes

ARScaleModel Virtual scale model augmented on table Stereo viewing using data glasses Virtual cursor for user interaction Including ARCatalogue for selecting components Virtual reality walk-through inside the model AR on the Table / Collaboration

ARScaleModel

Virtual scale model augmented on table

Stereo viewing using data glasses

Virtual cursor for user interaction

Including

ARCatalogue for selecting components

Virtual reality walk-through inside the model

Outdoors visualisation of Digitalo Commission by Senate Properties, spring 2003 Client/server implementation with PDA & WLAN Using large 80 x 80 cm markers, Speed 2 frames / sec Markerless implementation “AROnSite” Feature based tracking Interactive positioning of building to real scene E.g. Helsinki Music Hall visualisation 2006 Mobile Augmented Reality

Outdoors visualisation of Digitalo

Commission by Senate Properties, spring 2003

Client/server implementation with PDA & WLAN

Using large 80 x 80 cm markers, Speed 2 frames / sec

Markerless implementation “AROnSite”

Feature based tracking

Interactive positioning of building to real scene

E.g. Helsinki Music Hall visualisation 2006

Google Earth + AROnSite Building placed in Google Earth Semi-automatic positioning using GPS Applications, e.g. Paper mill visualisations by Pöyry Plc 2007 Excavator visualisations for Metso Minerals 2007 Future work Automatic positioning, e.g. matching map and cam view Implementation on camera phones “ Google Earth on Earth”

Google Earth + AROnSite

Building placed in Google Earth

Semi-automatic positioning using GPS

Applications, e.g.

Paper mill visualisations by Pöyry Plc 2007

Excavator visualisations for Metso Minerals 2007

Future work

Automatic positioning, e.g. matching map and cam view

Implementation on camera phones

ARWebCam Augmented reality webcamera Application during Digitalo construction 2003 – 2005 VTT being the first in the world to implement 3D Terrain Semiautomatic creation of 3D environment Based on aerial photos and geographic information Incl. trees, surrounding buildings, terrain shapes AR in the Web

ARWebCam

Augmented reality webcamera

Application during Digitalo construction 2003 – 2005

VTT being the first in the world to implement

3D Terrain

Semiautomatic creation of 3D environment

Based on aerial photos and geographic information

Incl. trees, surrounding buildings, terrain shapes

PRODUCTION APPLICATIONS

Industry Partners, e.g.

Further applications by Pöyry Plc ARCatalogue for technical machinery documents Helmet with integrated camera & data glasses Emagin video glasses / Micro-optical PC viewer Lightweight, battery mounted inside helmet Project SPECIAL

Further applications by Pöyry Plc

ARCatalogue for technical machinery documents

Helmet with integrated camera & data glasses

Emagin video glasses / Micro-optical PC viewer

Lightweight, battery mounted inside helmet

Objective To provide ubiquitous access to plant information using intuitive mobile user interfaces / Augmented Reality Results Core-task analyses for operational and maintenance work to consider and define interfaces between actions and actors including outsourcing of services An ontology based modeling environment for linking plant model information from several sources e.g. for augmented visualisation of 3D constructions AR demonstrations on gasification simulation process; the u ser can e.g. change controls and study changes in process values, and retrienve service instructions of the identified equipment Employing AR in the design phase to support information transfer between design, installation, implementation and maintenance Project PLAMOS

Objective

To provide ubiquitous access to plant information using intuitive mobile user interfaces / Augmented Reality

Results

Core-task analyses for operational and maintenance work to consider and define interfaces between actions and actors including outsourcing of services

An ontology based modeling environment for linking plant model information from several sources e.g. for augmented visualisation of 3D constructions

AR demonstrations on gasification simulation process; the u ser can e.g. change controls and study changes in process values, and retrienve service instructions of the identified equipment

Employing AR in the design phase to support information transfer between design, installation, implementation and maintenance

Objective: to study studies methods and tools for ubiquitous plant model usage Continuation to PLAMOS, but stronger emphasis on locationing techniques & pilot applications Location aware services and augmented reality techniques studied in process industry surroundings in order to provide plant model services to a mobile device Technologies: model based tracking, pseudosatellites Participating companies Fortum Nuclear Services, Pöyry Forest Industry, Metso Paper, Wärtsilä, Nokia Research Center, Cadmatic, Space Systems Finland, R.A. TR-Consulting Timetable 26.9.2007-31.7.2009, funding from Tekes and companies Project NOSE

Objective: to study studies methods and tools for ubiquitous plant model usage

Continuation to PLAMOS, but stronger emphasis on locationing techniques & pilot applications

Location aware services and augmented reality techniques studied in process industry surroundings in order to provide plant model services to a mobile device

Technologies: model based tracking, pseudosatellites

Participating companies

Fortum Nuclear Services, Pöyry Forest Industry, Metso Paper, Wärtsilä, Nokia Research Center, Cadmatic, Space Systems Finland, R.A. TR-Consulting

Timetable

26.9.2007-31.7.2009, funding from Tekes and companies

Augmented Assembly Timetable: 1.8.2006-31.7.2008 Funding from Tekes and companies Objective: Increasing efficiency in assembly work with Augmented Reality Development of the manufacturing process using AR Integration of AR to CAD/PLM/ERP systems Content creation (authoring) for Augmented Assembly Exploring AR display devices in assembly factory Work tool sensor feedback into AR systems Development of a AugAsse demonstration system Development of pilot applications at customer companies Advantages Visual and smart information for manual working phases Faster work and less faults Uniform quality Steep learning curve Customer-driven design change management Project AugAsse

Augmented Assembly

Timetable: 1.8.2006-31.7.2008

Funding from Tekes and companies

Objective: Increasing efficiency in assembly work with Augmented Reality

Development of the manufacturing process using AR

Integration of AR to CAD/PLM/ERP systems

Content creation (authoring) for Augmented Assembly

Exploring AR display devices in assembly factory

Work tool sensor feedback into AR systems

Development of a AugAsse demonstration system

Development of pilot applications at customer companies

Advantages

Visual and smart information for manual working phases

Faster work and less faults

Uniform quality

Steep learning curve

Customer-driven design change management

SOME NEW PROJECT IDEAS

Project Proposal ”TelePet” Internet Augmented Networked Games for Dogs To entertain dogs over Internet during master’s absence Game control based on gesture detection Example game Master throws virtual ball, dog has to “catch” Dog sees silhouette on the screen, ball bounces back if “caught” Further rewards, e.g. biscuit box operated over internet

Augmented Networked Games for Dogs

To entertain dogs over Internet during master’s absence

Game control based on gesture detection

Example game

Master throws virtual ball, dog has to “catch”

Dog sees silhouette on the screen, ball bounces back if “caught”

Further rewards, e.g. biscuit box operated over internet

Project Proposal ”AR4BC” “ Augmented Reality for Building and Construction” To provide mobile AR (MR) tools linking real situation at construction site to Building Information Models (BIM) information, planning and follow-up Augmented on-site visualisation linking project plans with actual situation on site Augmented site journaling feedback to system: attach images, tag installed parts in 4D model Office solutions presentation techniques for augmented 4D models

“ Augmented Reality for Building and Construction”

To provide mobile AR (MR) tools linking real situation at construction site to Building Information Models (BIM) information, planning and follow-up

Augmented on-site visualisation

linking project plans with actual situation on site

Augmented site journaling

feedback to system: attach images, tag installed parts in 4D model

Office solutions

presentation techniques for augmented 4D models

“ Mobile AR for Architects, Communities, Ads and Things” Technology – core solutions ARPhone – products, things downloaded by matrix codes, augmented in camera phone view ARMobile/AROnSite – products, things positioned in Google Earth maps, augmented in camera phone view ARCipher – 3D content augmented to ads, products etc., based on visual tracking of matrix codes Applications (“Things”) Consumer products, furniture, decoration Construction, houses, appartments Augmented ads, packages, products Entertainment, infotainment, games Fashion, trends, digital arts Personal digital items Virtual brand products Sharing by communities Project Proposal “MARACAT”

“ Mobile AR for Architects, Communities, Ads and Things”

Technology – core solutions

ARPhone – products, things downloaded by matrix codes, augmented in camera phone view

ARMobile/AROnSite – products, things positioned in Google Earth maps, augmented in camera phone view

ARCipher – 3D content augmented to ads, products etc., based on visual tracking of matrix codes

Applications (“Things”)

Consumer products, furniture, decoration

Construction, houses, appartments

Augmented ads, packages, products

Entertainment, infotainment, games

Fashion, trends, digital arts

Personal digital items

Virtual brand products

Sharing by communities

Connecting Physical and Digital Worlds Mobile Augmented Reality as visual interface to location based (3D) Internet content Internet content augmented in real world “ Google Earth on Earth” for displaying GE content on spot E.g. virtual ads, tourist info, navigation, architecture Mobile Internet content creation Augmented and other media on camera phones, e.g. photos, video clips Attaching the user generated (augmented) content to real world locations Interfaces to virtual products Digital products by virtual communities, with real monetary value Current examples Habbo Hotel, IRC Gallery, Aapeli, Second Life, … MAR interface to hyperdragging Interfaces to virtual worlds E.g. virtual world avatars augmented in real world And, real people, things augmented in virtual worlds

Mobile Augmented Reality as visual interface to location based (3D) Internet content

Internet content augmented in real world

“ Google Earth on Earth” for displaying GE content on spot

E.g. virtual ads, tourist info, navigation, architecture

Mobile Internet content creation

Augmented and other media on camera phones, e.g. photos, video clips

Attaching the user generated (augmented) content to real world locations

Interfaces to virtual products

Digital products by virtual communities, with real monetary value

Current examples Habbo Hotel, IRC Gallery, Aapeli, Second Life, …

MAR interface to hyperdragging

Interfaces to virtual worlds

E.g. virtual world avatars augmented in real world

And, real people, things augmented in virtual worlds

Mobile Augmented Reality one of the technologies “most likely to alter industries, fields of research, and the way we live” . - MIT’s annual review “10 Emerging Technologies 2007”

Technology Enablers and Drivers Smart mobile terminal Technology convergence to smart phones Incl. camera, 3D graphics, WLAN, WiMAX, 4G Up to accelerometers, RFID, 3D displays E.g. N95, iPhone, Samsung Minituriarised data glasses E.g. Lumus augmented reality glasses MyVu video glasses for iPod Game industry, e.g. Sony Eye Toy Nintendo Wii Eye of Judgment AR software technology E.g., Markerless tracking (see video ) Commercial activities, companies Intersense (USA), Metaio (Germany), Total Immersion (France)

Smart mobile terminal

Technology convergence to smart phones

Incl. camera, 3D graphics, WLAN, WiMAX, 4G

Up to accelerometers, RFID, 3D displays

E.g. N95, iPhone, Samsung

Minituriarised data glasses

E.g. Lumus augmented reality glasses

MyVu video glasses for iPod

Game industry, e.g.

Sony Eye Toy

Nintendo Wii

Eye of Judgment

AR software technology

E.g., Markerless tracking (see video )

Commercial activities, companies Intersense (USA), Metaio (Germany), Total Immersion (France)

Mixed Reality and 3D Web Second Life etc. Virtual communities and commerce Avatar representations & access from real world by Mixed Reality Google Earth Incl. 3D buildings, photos, panorama, etc. Content creation by virtual communities, “digital graffiti” Microsoft PhotoSynth, Massively multi-user content creation 3D reconstruction of physical world Nokia Purchase of Navteq at 5 Billion € MARA project & VCUI team work at Palo Alto 2007 IBM International Herald Tribune (April 25, 2007): IBM creates a mainframe platform for Virtual Worlds with game developer Hoplon, using the high-powered Cell processor, capable of supporting 100.000’s of users in realistic 3D games ...

Second Life etc.

Virtual communities and commerce

Avatar representations & access from real world by Mixed Reality

Google Earth

Incl. 3D buildings, photos, panorama, etc.

Content creation by virtual communities, “digital graffiti”

Microsoft

PhotoSynth, Massively multi-user content creation

3D reconstruction of physical world

Nokia

Purchase of Navteq at 5 Billion €

MARA project & VCUI team work at Palo Alto 2007

IBM

International Herald Tribune (April 25, 2007): IBM creates a mainframe platform for Virtual Worlds with game developer Hoplon, using the high-powered Cell processor, capable of supporting 100.000’s of users in realistic 3D games ...

Thank You! Prof. Charles Woodward Videos and Demos available at: http://www.vtt.fi/multimedia