3D Content Development and AR/VR Authoring

3D Content Development
and AR/VR Authoring
Part 2: AR/VR Authoring: Zi Siang See
International Society on Virtual Systems and MultiMedia 2017
(VSMM2017)
Research &
collaboration
Presentation: Zi Siang See
Research Workshop
Centre for Research-Creation in Digital Media
Zi Siang See
zisiangsee@sunway.edu.my
FB: www.facebook.com/zisiang
Research track: www.zisiangsee/research

UC Berkeley
Collaboration

Founding Projects
The Project

What to expect
Part 1
• 3D content for AR and VR
• Introduction to photogrammetry
Part 2
• AR and VR fundamental
• Basic AR authoring
• Demo on Unity + Vuforia
• Experience room-scale VR
Researcher: Zi Siang See

Photogrammetry

Photogrammetry
Paul Debevec
• PhD Thesis and contribution in photogrammetry (1996)
• High Dynamic Range (Debevec and Malik, 1997)
• Matrix bullet time effect
Paul Debevec in SIGGRAPH Asia, Light stage with multiple cameras configuration

Download this workshop content
http://www.su2crcdm.org/vsmm2017/layar_ar_sample.zip

User Experience: Quiver
Print, Color, Play, try it at http://www.quivervision.com

Is user interface, user experience
important to you?

Creative & Interactive Design
Business and Corporate Branding
Augmented Reality & Virtual Reality Research
Industrial Involvement since 2003/04

Research and Specialization
Professional Interest
• Creative Advertising
• Computational Photography
• Interactive Multimedia
• Virtual Reality 360 (VR360)
• Augmented Reality
• Imography Research
• Human Computer Interaction

Figure: Emphatic Computing Lab, University of South Australia

Introduction
Multimedia (Combination of Elements)
• Text
• Image
• Sound
• Video
• Animation
Industries
• Creative and advertising
• Corporate communications visuals
• Architectural imaging
• Cultural heritage preservation
• Scientific achieve

Introduction
Multimedia (Combination of Elements)
• Text
• Image
• Sound
• Video
• Animation
• Touch
• Scent
• Taste
• Sight
Industries
• Creative and advertising
• Corporate communications visuals
• Architectural imaging
• Cultural heritage preservation
• Scientific achieve
Development /Trend
• Augmented and Virtual Reality
• Human Computer Interaction

Figure: a vision about 2000 imagined in 1910, through 24 illustrations by Villemard.

Desk /Table Pocket /Palm /Hand Head /Eye Ubiquitous
Major changes in computing platforms

Presentation: Zi Siang SeePresentation: Zi Siang See

Figure: Super cockpit project, Tom Furness.

Virtual Reality
Figure: NASA, 1988 Virtual Environment System

Presentation: Zi Siang Seehttps://www.youtube.com/watch?v=NAuytnYU6JQ

Can we develop systems that
allow us to share what we are
seeing, hearing and feeling
with others?
(Empathic Computing, Mark Billinghurst)

Augmented Reality

How Google Glass Works?
• Google Glass, developer version, 2013
• Can be configured on user’s customized or personal spectacles, Android connectivity.
• Enterprise version available in 2017

Recon Jet
• Fitness glass
• Smartphone compatible
• Various use cases: solution-based applications

Presentation: Zi Siang Seehttps://www.youtube.com/watch?v=BhBtuNe_8is

• Displays, tracers, and AR systems in general need to become more accurate, lighter,
cheaper, and less power consuming (Azuma, 2001).
Figure: wearable computing, Steve Mann.

ideas worth spreading:
How can we develop AR and VR
content or systems for good
purposes?

Augmented, Virtual, Mixed Reality
• Effects of Sharing Physiological States of Players in a Collaborative Virtual Reality
Gameplay. (Dey et al., 2017)
• Digital Taste Interface project, Electronic taste stimulation (Ranasinghe et al., 2011)
• Billinghurst, M. (2011). The Future of Augmented Reality in Our Everyday Life.

Zi Siang See, Mohd Shahrizal Sunar, Mark Billinghurst, Arindam Dey, Delas Santano, Human Esmaeili, Harold Thwaites (2017)
Demo: Exhibition Approach Using an AR and VR Pillar. Siggraph Asia 2017 Mobile Graphics and Interactive Applications.
An Augmented Reality and Virtual Reality Pillar for Exhibitions
AR and VR Pillar Project
Zi Siang See, Mohd Shahrizal Sunar, Mark Billinghurst, Arindam Dey, Delas Santano, Human Esmaeili, Harold Thwaites (2017)
An Augmented Reality and Virtual Reality Pillar for Exhibitions: A Subjective Exploration. ICAT-EGVE.

Google Cardboard

Google Cardboard
Programmable NFC
Tag
Low cost resin
/plastic lens
Mobile device as
computing module
Endless configurable size and
shapes (and materials)
Magnetometer

Google Cardboard v2
Presses the
screen

What do I see?
http://www.zisiangsee.com/vtour/vtour_hdr_sphericalpanorama/index.html Presentation: Zi Siang See

Researcher: Zi Siang Seehttps://www.youtube.com/watch?v=hjKd24UCPYY

Samsung (Oculus) Gear VR
• VR /AR for end users
• Control interface included
• Social Media?
• UX prototyping
Figure: consumer product of HMD, suitable for AR and VR content

Gear 360

Gear 360
Lens 1 far from Lens 2
Lens 2 far from Lens 1
Nodal point is still not
centered, this can cause
serious parallax error
Figure: commercially available consumer 360 camera

360 Reproduction

AR and VR for Public Events
• User Experience is the focus - Distributable, medium of communications, believable
Figure: iLabs launching, 05 April 2017
Delas Santano, Zi Siang See, Chi How Fong, Harold Thwaites (2017) Aerial 360-Degree Video Research-Creation.
International Conference on Virtual Systems and Multimedia

Oculus
Figure: Oculus, by Facebook

Apple’s Recent Initiatives: ARKit and SteamVR Compatiblity
SteamVR Coming To Mac (unveiled in WWDC 2017), AR Kit will be available too.

Google Daydream
Figure: Android VR collaborating with other developers
Google has announced eight hardware partners that will make Daydream-ready phones,
including Samsung, HTC, LG, Xiaomi, Huawei, ZTE, Asus, and Alcatel.

Google Daydream
Google has announced eight hardware partners that will make Daydream-ready phones,
including Samsung, HTC, LG, Xiaomi, Huawei, ZTE, Asus, and Alcatel.

Steam VR
Presentation: Zi Siang SeeVinay Narayan - HTC Vive (https://visionsummit2017.com)

Researcher: Zi Siang Seewww.businessinsider.com/augmented-reality-in-the-enterprise-2017-6

Researcher: Zi Siang Seehttps://www.vrfocus.com/2017/06/500-new-vrar-jobs-to-be-created-as-nycedc-and-mome-announce-plan-for-immersive-lab
Office of Media and Entertainment (MOME)

User Experience
Presentation: Zi Siang Seehttp://www.aveva.com/en/Solutions/Product_Finder/AVEVA_AVP/

User Experience
Presentation: Zi Siang Seehttps://www.youtube.com/watch?v=sfowhrpS-6M

Presentation: Zi Siang Seehttps://www.youtube.com/watch?v=sfowhrpS-6M

Definitions
Virtual Reality
• Virtual Reality (VR) which can be referred to as immersive multimedia or computer-
simulated life, replicates an environment that simulates physical presence in places in the
real world or imagined worlds and lets the user interact in that world. Virtual reality
artificially creates sensory experiences, which can even include sight, hearing, touch, and
smell.
Augmented Reality
• Augmented Reality (AR) mixes a live real-world view with virtual interactive content on a
mobile or wearable device. One of the key enablers for this is tracking technology, such
as computer vision techniques for tracking off pre-defined markers or markerless images.

Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays, IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

AR + VR + MR = XR?

https://filmora.wondershare.com/virtual-reality/difference-between-vr-ar-mr.html

54% Want To Use VR/AR In Physical Stores
• 85% -- Travel experiences
• 84% -- Museums and galleries
• 84% -- Education, training
• 83% -- For designing something
• 79% -- Live entertainment
• 77% -- Exercise or sport
• 75% -- Navigation
• 74% -- Retail and shopping
• 70% -- Business
• 69% -- Meditation
• 63% -- Fully immersive gaming
• 53% -- Gaming through mobile
• Comprised interviews of 16,000 consumers.
• Consumer's perspective: the main reason for not having tried VR or AR is the lack of
opportunity, based on a new study.
https://www.mediapost.com/publications/article/301863/54-want-to-use-vrar-in-physical-stores.html

Gartner’s Hype Cycle

Through Heart and Eyes:
Sharing what you feel and
interacting with what you see
(Thammathip Piumsomboon, 2017)

Brings value to the community

Figure: Quiver, the 3D Augmented Reality coloring book technology, originally called colAR, was developed at the HIT Lab NZ.

Empathy Glasses
Katsutoshi Masai, Kai Kunze, Maki sugimoto, Mark Billinghurst (2016) Empathy Glasses.
Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

Presentation: Zi Siang Seehttps://www.youtube.com/watch?v=CdgWVDbMwp4

Practical consideration
• Emphatic computing
• Shared experience, emotion, same viewpoint
• Real-time
• “Put myself in their shoes”
• Achieve and complete tasks together

Hair Raising Message

Presentation: Zi Siang Seehttps://www.youtube.com/watch?v=064ipuBiWDg

• Ubiquitous computing – the installation of hardware are invisible to users
• Focus on user experience
• Has strong storyline
• Brings social good and awareness

Penguin Navi

Presentation: Zi Siang Seehttps://www.youtube.com/watch?v=IK4-zPD_25U

• Mix computing and entertainment
• High level of usability design
• Focus on bringing the intended outcome – going to the aquarium
• Make use of existing mobile devices and capability

Huggy Pajama
James Keng Soon Teh, Adrian David Cheok, Roshan L. Peiris, Yongsoon Choi, Vuong Thuong, Sha Lai (2008) Huggy Pajama: a mobile parent and child hugging communication system.
Proceedings of the 7th international conference on Interaction design and children.
Presentation: Zi Siang Seehttps://www.youtube.com/watch?v=hQ6usrx-GPM

Mobile Audible AR
Zi Siang See, Mark Billinghurst, Vimalan Rengganaten, Shelly Soo (2016) Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality.
Siggraph Asia 2016 Mobile Graphics and Interactive Application

Mobile Audible AR
https://www.youtube.com/watch?v=VGsaTDKC2qY

Virtual Reality in Medicine Leadning
University of Basel, Switzerland
https://www.unibas.ch/en/News-Events/News/Uni-Research/Virtual-Reality-in-Medicine.html

Presentation: Zi Siang Seehttps://youtu.be/AttXbcLUyR0

VRPhysio

VRPhysio
• Customizable
• Clinic can track and help patients remotely
• Analytics

Presentation: Zi Siang Seehttps://www.youtube.com/watch?v=WmSyWb6djDU

Research Possibilities
Exploration
• Method and apparatus for augmented reality
• Method and apparatus for virtual reality
• Ubiquitous computing
• Obstacles and issues
• Technology management
• Education technology
• Engineering and concept design
• User interface (UI) design
• User experience (UX)

Can we use VR/AR technologies to help more people?
Effort
• Empathic computing
• Experience synchronization
• Improve well-being
• Medical-based benefits
• Archeology, cultural heritage: see the whole story in 360 (digital preservation)
• Sharing community and social stories
• Combine interest of different professional disciplines

Some on-going Projects

“Creativity is not an option, it’s an absolute necessity”
Sir Ken Robinson 2014

Figure: European researchers used 3D-modeling and Instagram to recreate Mosul Museum destroyed in civil war.

VR brings ninth century Viking invaders' camp to life (YorkShire Museum, 2017) - staged in partnership with the British Museum, new research by the universities of York and Sheffield.
https://www.theguardian.com/artanddesign/2017/may/17/virtual-reality-brings-ninth-century-viking-invaders-camp-to-life

Figure: VSMM2016 also features the system by Prof Sarah Kenderdine

Figure: VSMM2016 featured the system by Prof Sarah Kenderdine

Figure: VSMM2016 featured the system by Prof Sarah Kenderdine
Pureland: Inside the Mogao Grottoes at Dunhuang

Building Information Modeling VR
On-going Project Development
• FARO 3D Scanning (University of Reading)
• VR with freedom of room-scale locomotion (Sunway University)
Research
collaboration

VR for Disaster Relief
• Asia-Pacific SpatioTemporal Institute (Taiwan)
• Location-based VR visualization
Research
collaboration
Semi-automated
post-processing
Acquisition and
Technical Setup
User experience
/Distribution
Disaster sites:
- scenario?
- consideration?
UC Berkeley
ADVANCED DIGITAL IMAGING FOR
PRESERVATION, CONSERVATION
AND EXHIBITION DESIGN
SCHEDULE. 11th
– 13th
SEPTEMBER 2013
WEDNESDAY 11th
SEPTEMBER
9:30 – 9:40 Welcome to workshop DR. SARAH
KENDERDINE(SK)
9:40 – 10:00 Welcome to HKSTP, ALiVE& LUXLAB
10:00 – 10:15 Outline of workshop
10:15 – 11:15 Demonstration of optical scanning DR. JAYARRETOQUE
(JA)
Ms. Mie KADO,
Mr Kouji OKUMURA
11:15 – 12:30 Introduction to immersive museums SK
12:30 – 1:30 Highlights fromoptical scanning projects DR. JAYARRETOQUE
1:30 – 2:30 Lunch
2:30 – 3:30 Tour of ALiVEexhibitions DR. KENDERDINE
3:30 – 5.00 Highlights fromoptical scanning projects PROF. ARI IDE-
EKTEBASSI (AIE)
5:00 – 5:30 Q& A TEAM
5:30 – 6:30 Transfer to Diamond Hill by bus OPTIONAL(please
register with Kyle Chung
for catering purposes
before 1st September)
6:30 – 7:30 Tour of NamLian Gardens < nlgarden.org>
7:30 – 9:00 Vegetarian banquet at the Chi Lin Restaurant at
NLG(HK$450 & *students HK$225 per head)
THURSDAY 12th
SEPTEMBER
9:30 – 10:30 Ultra high resolution and high colour definition AIE
10:30 – 12:00 Analytical Imaging I: Introduction to Analytical
Imaging
JA
12:00 – 12:30 Introduction to Hong Kong Maritime Museum
project
MR. ROBERTTRIO
12:30 – 1:00 Q& A TEAM
1:00 – 2:00 Lunch
2:00 – 4:30 Scanning and training JA
4:30 – 6:00 Tour of Creative Media Centre exhibitions
including the Pure Land projects (optional).
Proceed to venue by taxi.
SK
Asia-Paciﬁc SpatioTemporal Institute
UC Berkeley
INTERNATIONAL WORKSHOP ON
ADVANCED DIGITAL IMAGING FOR
PRESERVATION, CONSERVATION
AND EXHIBITION DESIGN
SCHEDULE. 11th
– 13th
SEPTEMBER 2013
WEDNESDAY 11th
SEPTEMBER
KENDERDINE(SK)
(JA)
Ms. Mie KADO,
Mr Kouji OKUMURA
1:30 – 2:30 Lunch
EKTEBASSI (AIE)
5:00 – 5:30 Q& A TEAM
THURSDAY 12th
SEPTEMBER
Imaging
JA
project
MR. ROBERTTRIO
12:30 – 1:00 Q& A TEAM
1:00 – 2:00 Lunch
SK

Virtual Reality Cultural Heritage
• Electronic Cultural Atlas Initiative (ECAI) & CRCDM
• Bujang Valley - digital preservation research-creation
Research
collaboration
UC Berkeley
KENDERDINE(SK)
(JA)
Ms. Mie KADO,
Mr Kouji OKUMURA
1:30 – 2:30 Lunch
EKTEBASSI (AIE)
5:00 – 5:30 Q& A TEAM
THURSDAY 12th
SEPTEMBER
Imaging
JA
project
MR. ROBERTTRIO
12:30 – 1:00 Q& A TEAM
1:00 – 2:00 Lunch
SK

Museum VR: ECAI
• Electronic Cultural Atlas Initiative
• User experience and digital preservation of cultural artifacts

Museum VR: The Tomb of Sultan
• 3D Scanned /Photogrammetry of artifacts
• User experience –revisiting an ancient site where the king rested.

Museum VR: The Tomb of Sultan – Interaction?
• Can we see the hands? And can we gain some control? Physics?

Museum VR: The Tomb of Sultan – Interaction?
• Hands VR assisted by Paul Tschirgi (2017), allowed to be customizable.

Authoring Basics

Download this workshop content
http://www.su2crcdm.org/vsmm2017/layar_ar_sample.zip

Start here: https://www.layar.com

Sign up for a developer account: https://www.layar.com/accounts/register

Login to Layar Creator
We can see that the simple interface allows us
to create “NEW CAMPAIGN” and manage our
developer account.

Create “NEW CAMPAIGN”
Provide a unique project name, you can call it
anything that matches your content or ideas.
For demonstration, we choose “Magazine” for
our setting, and then create a new campaign.

Create “NEW CAMPAIGN”
To begin, we need to have “pages” which
allows content to be tracked. We usually call
these image-based tracker an “AR marker”.

ar001-004 sample images are provided for
quick development and testing, use these
images as content you want to be
augmented.
You may consider to move these files to any
new folder based of your project.
marker001-004 sample marker images are
provided for quick development and testing.
These images are to be tracked by your AR
application and AR content being augmented
on top of them.
To do: you may also want to print these images
out so that you can test it as you develop you
content.

marker001-004: select and drag these tracker
images to Layar’s website where you can “ADD
NEW PAGES”.

Drag all the “tracker” sample images into
“ADD NEW PAGES” area.
This is where you have sample pages of your
magazine prototype.

ADD 4 PAGES
Once all “markers” images are accepted and
uploaded, proceed with adding these pages.
Note: we have to think that these are pages of
our sample magazine

Pages are being analyzed and uploaded as we
can see. Once completed we can start to add
tons of features on it.

Start Editing!
We can now add AR features

Select “Basic”
We can begin with having “SHOW
AN IMAGE” feature on top of this
magazine sample cover.

SHOW AN IMAGE
You will see that a template is
being prepared and instruction says
we can drag an image or upload an
image here.

SHOW AN IMAGE
Drag the image “ar001” from the
sample folder here to be uploaded.
Once okay, click save.

SHOW AN IMAGE
Save to proceed.

Reposition and scale your AR image
You can scale, reposition your ar
image to be placed at a suitable
location as you prefer.

Repeat the process for tracker 002-004
Process can be repeated for tracker002
to 004 as provided in the sample folder.
Of course, you might want to try some
features from media, social and
advanced too.

Example: adding video
We may also copy URL from youtube
and include it into our project

Mixing multiple AR content
We may also mix multiple ar content
such as a map image (provided in the
sample folder) and a facebook button.

Testing

TEST
Before we decided to publish (which
requires a cost from platform provider),
click on the “Test” tab which we can
experiment with full feature.
However we may not be able to share
the content unless we get it published.

TEST YOUR CAMPAIGN
As we follow the instruction. We should
have a Layar app installed in our mobile
or tablet device. Then scan the “page”
as we follow the on-screen instruction
on the mobile device. Then we should
be seeing the samples working.
You may also want to print these
samples to see it working when you
have a mockup magazine.

AR Content
loaded!
Tap Screen

Multiple
AR contentImage-based
AR content
Video AR
content

Pro and Cons: using portal-based development
Advantages
• Easy for quick prototyping
• Free for testing
• Content hosted on provider’s server
Disadvantages
• Minimum customization (eg. lack of 3D)
• Rely on the provider’s app and brand
• Less flexible payment method or costly for publishing

Ideas for AR? Start building yours :)

Demo: Unity + Vuforia

What preparations do we need?
A Set of Usable Markers (Trackers)
• Image-based
• 3D
• Hybrid approach?
Configuration
• Unity
• Vuforia SDK
• Mobile or tablet device

What preparations do we need?
A Set of Usable Markers (Trackers)
• Image-based
• 3D
• Hybrid approach?
Configuration
• Unity
• Vuforia SDK
• Mobile or tablet device
Are these images suitable markers?

3D content from photogrammetry?
.fbx
.obj

Thank you
Research Workshop Seminar
Centre for Research-Creation in Digital Media

MOOC:
Virtual Reality, Augmented Reality:
The Promise of Sci-Fi?
Massive Open Online Course: AR and VR
Preparation: Zi Siang See
Center for Research-Creation in Digital Media
https://www.openlearning.com/sunway-university

Additional

VR Experiences: Mobile, Room-scale, World-scale

Researcher: Zi Siang Seehttps://www.youtube.com/watch?v=67yLuiSfMWM

User Experience

User Experience
Researcher: Zi Siang See•https://www.youtube.com/watch?v=YJg02ivYzSs

Use Cases: Ikea AR

Researcher: Zi Siang See•https://www.youtube.com/watch?v=vDNzTasuYEw

Use Cases: McDonald’s

Researcher: Zi Siang Seehttps://www.youtube.com/watch?v=laIqVPTXT88

User Experience: Octagon 4D
Occupation Series Animal Series

Researcher: Zi Siang Seehttps://www.youtube.com/watch?v=YKWI4Lbrnj8

Virtual Monitors, Navigation
Figure: over 10,000 surgeries have been successfully completed using Scopis technology and the Navigation Platform has a lot of potential for use beyond spinal surgery.
https://uploadvr.com/berlin-company-using-hololens-improve-surgical-procedures/
Presentation: Zi Siang Seehttps://youtu.be/xvbWE4OsKxY?list=PLCHEesPXRgF74fGSPd6SK-mda35E32up-

Virtual Monitors, Navigation
https://youtu.be/xvbWE4OsKxY?list=PLCHEesPXRgF74fGSPd6SK-mda35E32up-

Input

Researcher: Zi Siang Seehttps://www.youtube.com/watch?v=bhA4PBns0Mo

In the future, we see people
interacting with the virtual world and
real world in seamless, frictionless
and continuous ways, not bound by
delineations of experience between
“VR,” “AR” or “MR.”
(snap, 2017)
https://techcrunch.com/2017/05/02/xr-a-new-way-to-consider-the-reality-continuum/

Empathic Computing:
Systems that create and share
understanding
(Mark Billinghurst)

See you again :)
www.zisiangsee.com/portfolio
www.zisiangsee.com/research

3D Content Development and AR/VR Authoring

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 3D Content Development and AR/VR Authoring

Similar to 3D Content Development and AR/VR Authoring (20)

Recently uploaded

Recently uploaded (20)

3D Content Development and AR/VR Authoring