This document discusses various techniques for prototyping augmented reality interfaces, including sketching, storyboarding, wireframing, mockups, and video prototyping. Low-fidelity techniques like sketching and paper prototyping allow for rapid iteration and exploring interactions at early stages. Higher-fidelity techniques like interactive mockups and video prototypes communicate the look and feel of the final product and allow for user testing. A variety of tools are presented for different stages of prototyping, from sketching and interactive modeling in VR, to scene assembly using drag-and-drop tools, to final mockups using design software. Case studies demonstrate applying these techniques from initial concepts through to higher-fidelity prototypes. Overall the document

1. AR PROTOTYPING
COMP 4010 Lecture Five
Mark Billinghurst
August 25th 2022
mark.billinghurst@unisa.edu.au

2. REVIEW

3. How Do You Design an Interface for This?

4. Different Types of AR Interaction
• Browsing Interfaces
• simple (conceptually!), unobtrusive
• 3D AR Interfaces
• expressive, creative, require attention
• Tangible Interfaces
• Embedded into conventional environments
• Tangible AR
• Combines TUI input + AR display

5. 1. AR Interfaces as Data Browsers
• 2D/3D virtual objects are
registered in 3D
• “VR in Real World”
• Interaction
• 2D/3D virtual viewpoint control
• Applications
• Visualization, training

6. Example: Google Maps AR Mode
• AR Navigation Aid
• GPS + compass, 2D/3D object placement

7. 2. 3D AR Interfaces
• Virtual objects displayed in 3D
physical space and manipulated
• HMDs and 6DOF head-tracking
• 6DOF hand trackers for input
• Interaction
• Viewpoint control
• Traditional 3D user interface
interaction: manipulation, selection,
etc.
Kiyokawa, et al. 2000

8. 3. Augmented Surfaces and Tangible Interfaces
• Basic principles
• Virtual images are projected
on a surface
• Physical objects are used as
controls for virtual objects
• Support for collaboration
Wellner, P. (1993). Interacting with paper on the
DigitalDesk. Communications of the ACM, 36(7), 87-96.

9. Tangible User Interfaces (Ishii 97)
• Create digital shadows
for physical objects
• Foreground
• graspable UI
• Background
• ambient interfaces

10. i/O Brush (Ryokai, Marti, Ishii) - 2004
Ryokai, K., Marti, S., & Ishii, H. (2004, April). I/O brush: drawing with everyday objects as ink.
In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 303-310).

11. Lessons from Tangible Interfaces
• Physical objects make us smart
• Norman’s “Things that Make Us Smart”
• encode affordances, constraints
• Objects aid collaboration
• establish shared meaning
• Objects increase understanding
• serve as cognitive artifacts

12. But There are TUI Limitations
• Difficult to change object properties
• can’t tell state of digital data
• Limited display capabilities
• projection screen = 2D
• dependent on physical display surface
• Separation between object and display
• ARgroove – Interact on table, look at screen

13. Orthogonal Nature of Interfaces
3D AR interfaces Tangible Interfaces
Spatial Gap No – Interaction is
Everywhere
Yes – Interaction is
only on 2D surfaces
Interaction Gap
Yes – separate
devices for physical
and virtual objects
No – same devices for
physical and virtual
objects

14. 4. Tangible AR: Back to the Real World
• AR overcomes display limitation of TUIs
• enhance display possibilities
• merge task/display space
• provide public and private views
• TUI + AR = Tangible AR
• Apply TUI methods to AR interface design
Billinghurst, M., Kato, H., & Poupyrev, I. (2008). Tangible augmented reality. ACM Siggraph Asia, 7(2), 1-10.

15. Tangible AR: Tiles (Space Multiplexed)
• Tiles semantics
• data tiles
• operation tiles
• Operation on tiles
• proximity
• spatial arrangements
• space-multiplexed
Poupyrev, I., Tan, D. S., Billinghurst, M., Kato, H., Regenbrecht, H., & Tetsutani, N. (2001,
July). Tiles: A Mixed Reality Authoring Interface. In Interact (Vol. 1, pp. 334-341).

16. Tangible AR: Time-multiplexed Interaction
• Use of natural physical object manipulations to control
virtual objects
• VOMAR Demo
• Catalog book:
• Turn over the page
• Paddle operation:
• Push, shake, incline, hit, scoop
Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000, October). Virtual object manipulation on a table-top AR
environment. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000) (pp. 111-119). IEEE.

17. 5. Natural AR Interfaces
• Goal:
• Interact with AR content the same
way we interact in the real world
• Using natural user input
• Body motion
• Gesture
• Gaze
• Speech
• Input recognition
• Nature gestures, gaze
• Multimodal input
FingARtips (2004)
Tinmith (2001)

18. Evolution of AR Interfaces
Tangible AR
Tangible input
AR overlay
Direct interaction
Natural AR
Freehand gesture
Speech, gaze
Tangible UI
Augmented surfaces
Object interaction
Familiar controllers
Indirect interaction
3D AR
3D UI
Dedicated
controllers
Custom devices
Browsing
Simple input
Viewpoint control
Expressiveness, Intuitiveness

19. DESIGNING AR INTERFACES

20. Interaction Design
“Designing interactive products to support
people in their everyday and working lives”
Preece, J., (2002). Interaction Design
• Design of User Experience with Technology

21. Typical Interaction Design Cycle
Develop alternative prototypes/concepts and compare them, And iterate, iterate, iterate....

22. PROTOTYPING

23. Example: Google Glass

24. Tom Chi’s Prototyping Rules
1. Find the quickest path to experience
2. Doing is the best kind of thinking
3. Use materials that move at the speed of
thought to maximize your rate of learning

25. How can we quickly prototype
XR experiences with little or no
coding?

26. Prototyping in Interaction Design
Key Prototyping
Steps

27. ● Quick visual design
● Capture key interactions
● Focus on user experience
● Communicate design ideas
● “Learn by doing/experiencing”
Why Prototype?

28. Importance of Physical and Digital Prototypes
• Quick & dirty
• Explore interactions
• Get initial user feedback
• Avoid premature commitment
• Devise technical requirements

29. Typical Development Steps
● Sketching
● Storyboards
● UI Mockups
● Interaction Flows
● Video Prototypes
● Interactive Prototypes
● Final Native Application
Increased
Fidelity and
Interactivity

30. From Idea to Product
Define
Requirements
CityViewAR
1
2
3
4
5
6
1
Sketch
Interface
2
Rough
Wireframes
3
Interactive
Prototype
4
High Fidelity
Prototype
5
Developer
Coding
6
User
Testing
7
Deploy
App
8
Develop
Iterate

31. From Idea to Product

32. XR Prototyping Techniques
Lo-
Fi
Hi-
Fi
Easy
Hard
Digital
Authoring
Immersive
Authoring
Web-Based
Development*
Cross-Platform
Development*
Native
Development*
* requires scripting and 3D programming skills
Sketching
Paper Prototyping
Video Prototyping
Wireframing
Bodystorming
Wizard of Oz

33. XR Prototyping Tools
Low Fidelity (Concept, visual design)
• Sketching
• Photoshop
• PowerPoint
• Video
High Fidelity (Interaction, experience design)
• Interactive sketching
• Desktop & on-device authoring
• Immersive authoring & visual scripting
• XR development toolkits

34. Advantages/Disadvantages
Prototype Advantages Disadvantages
Low-fidelity
prototype
- low developmental cost
- evaluate multiple
design concepts
- limited error checking
- navigational and flow
limitations
High-fidelity
prototype
- fully interactive
- look and feel of final
product
- clearly defined
navigational scheme
- more expensive to develop
- time consuming to build
- developers are reluctant to
change something they have
crafted for hours

35. From Sketch to Product
•Sketches:
- early ideation stages of design
•Prototypes:
- detailing the actual design

36. Content and Interaction
Polished
Content
Placeholder
Content
Implicit & Explicit
Interactions
“click”
Lo-Fi Hi-Fi

37. XR Prototyping Tools
Low Fidelity (Concept, visual design)
• Sketching
• Photoshop
• PowerPoint
• Video
High Fidelity (Interaction, experience design)
• Interactive sketching
• Desktop & on-device authoring
• Immersive authoring & visual scripting
• XR development toolkits

38. From Sketch to Prototype
Storyboard Wireframe Mock-up
Sketch Interactive
Prototype

39. Your Most Valuable Prototyping Tool..

40. Interface Sketching

42. Sketching as Dialogue
“Sketching is about the Activity not the Result” - Bill Buxton

43. Buxton’s Key Attributes of Sketching
• Quick
• Work at speed of thought
• Timely
• Always available
• Disposable
• Inexpensive, little investment
• Plentiful
• Easy to iterate
• A catalyst
• Evokes conversations

44. Sketching Resources

45. AR/VR Interface Design Sketches
•Sketch out Design concept(s)

48. AR Augmented Sketch
● Sketch on top of photos of real world – target workplaces
Luisa Vasquez - https://blog.prototypr.io/10-tips-for-rapid-prototyping-on-the-hololens-2-bda021e21743

49. Bodystorming - Generating Ideas
Using physical props to generate ideas for AR/VR experiences

50. Acting out the User Experience
Walking through an AR/VR experience with physical objects

51. Storyboarding - Describing the Experience
http://monamishra.com/projects/Argo
.html

52. Key Elements
1. Scenario: Storyboards are based on a scenario or a user story. The
persona or role that corresponds to that scenario is clearly specified
2. Visuals: Each step in the scenario is represented visually in a sequence.
The steps can be sketches, illustrations, or photos.
3. Captions: Each visual has a corresponding caption. The caption
describes the user’s actions, environment, emotional state, device, etc.

53. Storyboard
Template

54. https://smol_inklings.artstation.com/projects/XBr8b3

55. Low and High Fidelity Storyboards
Low fidelity
High fidelity

56. Storyboard Guidelines
Successful Storyboards:
§ Visually tell the story
§ Detail the concept, characters, setting, sequence, interactions
§ Reflect key ideas and POV
Describe the following:
§ Field-of-View: What surroundings can the viewer see?
§ Estimated POV: Where is viewer positioned in their surroundings?
§ Key Frames: What is the starting and ending points of the scene?
§ Interaction: How does the viewer interact in this scene?

57. Wireframes
It’s about
- Functional specs
- Navigation and interaction
- Functionality and layout
- How interface elements work together
- Defining the interaction flow/experience
Leaving room for the design to be created

58. Task Flow
User flow with the application
Informs screen/interface elements

59. Example Wireframe

60. Mockup
It’s about
- Look and feel
- Building on wireframe
- High fidelity visuals
- Putting together final assets
- Getting feedback on design

64. Interface Sketching in VR
Using VR applications for rapid prototyping
- Intuitive sketching in immersive space
- Creating/testing at 1:1 scale
- Rapid UI design/layout
Examples
- Quill - https://quill.fb.com/
- Tilt Brush - https://www.tiltbrush.com/

65. Example: VR Concept Interface for SketchUp
https://www.youtube.com/watch?v=5FR3BLkgSPk

66. Designing AR in VR
https://www.youtube.com/watch?v=TfQJhSJQiaU

67. Demo
https://www.youtube.com/watch?v=TfQJhSJQiaU

68. Scene Assembly In AR
• Many tools for creating AR scenes
• Drag and drop your assets
• Develop on web, publish to mobile
• Examples
• Catchoom - CraftAR
• Blippar - Blipbuilder
• ARloopa - Arloopa studio
• Wikitude - Wikitude studio
• Zappar - ZapWorks Designer

69. CraftAR
•Web-based AR marker tracking
•Add 3D models, video, images to real print content
•Simple drag and drop interface
•Cloud based image recognition
•https://catchoom.com/augmented-reality-craftar/

70. CraftAR Demo
•https://www.youtube.com/watch?v=f42MqLF5Odw

71. Vuforia Studio
• Author animated AR experiences
• Drag and drop content
• Add animations
• Import CAD models
• Combine with IOT sensors
• https://www.ptc.com/en/products/vuforia/vuforia-studio

72. https://www.youtube.com/watch?v=0kfIKMOqAPc

73. Mock-up Guidelines
1. Generate final 2D/3D interface elements
2. Replace wireframe UI elements with high quality visuals
3. Use standard AR/VR UI elements
4. Simulate AR/VR views
5. Focus on visual/audio design
6. Collect feedback from target end-users

74. Sketch vs. Wireframe vs. Mock-up
Low Fidelity Low to Medium
Fidelity
Medium to High Fidelity
IDEATE FLOW VISUALIZE

76. Adding Transitions

77. Paper Prototyping

78. Basic Materials
● Post-its
● 5x8 in. index cards
● Scissors, X-acto knives
● Overhead transparencies
● Large, heavy, white paper (11 x 17)
● Tape, stick glue, correction tape
● Pens & markers (many colors & sizes)

79. Example: Mobile Paper Prototype
https://www.youtube.com/watch?v=jTytI1PkGFM

80. AR Prototyping with Layers
● Separate world-stabilized and head stabilized
○ Draw world stabilized on background paper
○ Draw head stabilized on transparent plastic
● Simulate Field of View of AR HMD
Lauber, F., Böttcher, C., & Butz, A. (2014, September). Papar: Paper prototyping for augmented reality. In Adjunct Proceedings
of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 1-6).
FOV in Red
Head Stabilized - Foreground
World Stabilized - Background

81. Example: Mobile AR Prototyping
https://www.youtube.com/watch?v=Hrbflbct99o

82. AR Visualization with Props
https://www.youtube.com/watch?v=dXBXBHLqrLM

83. ShapesXR - www.shapesxr.com/
Immersive VR tool for:
● Design
● Prototyping
● Communication
Key Features
● Assets Library and Importing
● Storyboarding
● Multi-user collaboration
● Export and Sharing

84. Shapes XR Workflow

85. https://www.youtube.com/watch?v=zmjL7PsU4eE
Shapes XR Demo

86. Prototyping Gesture Interface
https://www.youtube.com/watch?v=A_uWdVNNYjo

87. Prototyping AR View
https://www.youtube.com/watch?v=NzA6YB3wJlA

88. Case Study:
Higher Fidelity Prototype
https://medium.com/@dylanhongtech/augmented-reality-prototyping-tools-for-head-mounted-displays-c1ee8eaa0783

89. AR Prototyping for HMDs
•Tools
•Storyboarding - Concept
•Sketch – high fidelity 2D
•Cinema4D – 3D animation
•SketchBox – 3D layout
•Photoshop – Visual mockup
•PowerPoint - Interactivity

90. Sketch Design
•User Interface Layout
•Overlay AR elements on blurry background

91. Cinema4D
•3D animation tool
•Interface element mock-up

92. SketchBox
•3D user interface layout

93. Photoshop
•High end visual mock-ups
•Add AR content to real world views

94. Unity/MRTK Prototyping
•Viewing on the Hololens2, simple gesture interaction

95. Final PowerPoint Prototype
•Simple interactive demo

96. Video Sketching
• Process
• Capture elements of real world
• Use series of still photos/sketches in a movie format.
• Act out using the product
• Benefits
• Demonstrates the product experience
• Discover where concept needs fleshing out.
• Communicate experience and interface
• You can use whatever tools you want, e.g. iMovie.

97. Example ToastAR (Apple)
View here: https://developer.apple.com/videos/play/wwdc2018/808/

98. AR Video Sketching
https://www.youtube.com/watch?v=vityu-IgHLQ

100. Google Glass Concept Movie
https://www.youtube.com/watch?v=5R1snVxGNVs

101. https://www.youtube.com/watch?v=gg-ZakMEwDU

102. Example: Sketch to Video

109. https://www.youtube.com/watch?v=5hRq0BRn22M

110. MOBILE AUGMENTED
REALITY FOR SPATIAL
NAVIGATION
Sharon Brosnan
0651869
Bachelor of Science in Digital Media Design

111. Bunratty Folk Park

112. Project Goals
Make a mobile AR Navigational Aid
○ Easy to use, clear and understandable
○ Useful to visitors
○ Creating interaction between the visitor and the aid through the user interface
○ Engage the visitor
To ensure this…
○ It is necessary to understand the visitor of a navigational aid in Bunratty Folk Park
○ Identify visitor motives and goals while going through the Folk Park.

113. NeedsAnalysis
Understanding Technology and Related Work
• Literature
• Similar Technologies
• Electronic Tours in Museum Settings
• Interactions design in Outdoor Museums
Understanding the User over time
• Observations
• Interviews

114. Findings
• Most visitors do not use the map
• Most visitors have mobile phones
• Visitors want more information
• View the Folk Park at their own pace
• Information should be straight to the point
• Large social interaction within groups
Needs Analysis

115. Desired Functionality
• View Options
• CameraView
• MapView
• ListView
• Sub-Options
• Places
• Events
• Restaurants
• Augmented Reality Features
for navigation
• Text Information
• 3D Objects
• 3D Tour Guide
• 3D Placement of Buildings

116. Storyboard

117. ITERATIVE DESIGN PROCESS
Prototyping and UserTesting
• Low Fidelity Prototyping
• Sketches
• Paper Prototyping
• Post-It Prototyping
• PowerPoint Prototyping
• High Fidelity Prototyping
• Wikitude

118. INITIAL SKETCHES (Photoshop)
Pros:
• Good for idea generation
• Cheap
• Concepts seem feasible
Cons:
• Not great feedback gained
• Photoshop not fast enough for
making changes

119. Post-it Prototyping

120. POWERPOINT PROTOTYPING
Benefits
• Used for User Testing
• Interactive
• Functionalities work when following the
story of Scenario 1
• Quick
• Easy arrangement of slides
User Testing
• Participants found
• 15 minute sessions screen captured
• ‘Talk Allowed’ technique used
• Notes taken
• Post-Interview

121. WIKITUDE
• Popular augmented reality browser
for mobile devices
• Mapping
• Point of Interest abilities
• Multiplatform
• Shows the points of interest of
Bunratty Folk Park
• Markers can be selected in and an
information pop-up appears

122. VIDEO PROTOTYPE
¢ Flexible tool for capturing the use
of an interface
¢ Elaborate simulation of how the
navigational aid will work
¢ Does not need to be realistic in
every detail
¢ Gives a good idea of how the
finished system will work

123. DemoVideo
• https://www.youtube.com/watch?v=8BqbEXUyDfo

124. PROTOTYPING INTERACTIONS

125. XR Prototyping Tools
Low Fidelity (Concept, visual design)
• Sketching
• Photoshop
• PowerPoint
• Video
High Fidelity (Interaction, experience design)
• Interactive sketching
• Desktop & on-device authoring
• Immersive authoring & visual scripting
• XR development toolkits

126. XR Tools Landscape
Digital & Immersive Authoring
Proto.io, Tour Creator, ...
Tilt Brush, Blocks, Quill, …
Web-Based Development
THREE.js, Babylon.js, …
A-Frame, AR.js, …
Cross-Platform Development
Unity + SDKs
Unreal + SDKs
Native Development
Cardboard/Oculus/Vive/... SDK
Vuforia/ARCore/ARKit/… SDK

127. XR Tools Landscape
Digital & Immersive Authoring
Good for storyboarding but limited
support for interactions
Web-Based Development
Good for basic XR apps but often
interactions feel unfinished
Native Development
Good for full-fledged XR apps but
limited to a particular platform
Cross-Platform Development
Good for full-fledged XR apps but
usually high learning curve

128. XR Prototyping Techniques
Lo-
Fi
Hi-
Fi
Easy
Hard
Digital
Authoring
Immersive
Authoring
Web-Based
Development*
Cross-Platform
Development*
Native
Development*
* requires scripting and 3D programming skills
Sketching
Paper Prototyping
Video Prototyping
Wireframing
Bodystorming
Wizard of Oz

129. Digital Prototyping
Lo-
Fi
Hi-
Fi
Easy
Hard
Digital
Authoring
Immersive
Authoring
Web-Based
Development*
Cross-Platform
Development*
Native
Development*
* requires scripting and 3D programming skills

130. Digital Authoring Tools for AR
Vuforia Studio
Lens Studio
• Support visual authoring of marker-
based and/or marker-less AR apps
• Provide default markers and support
for custom markers
• Typically enable AR previews
through emulator but need to deploy
to AR device for testing

131. Immersive Authoring Tools for AR
• Enable visual authoring of 3D
content in AR
• Make it possible to edit while
previewing AR experience in the
environment
• Provide basic support for interactive
behaviors
• Sometimes support export to
WebXR
Apple Reality Composer
Adobe Aero

132. Interactive Sketching
•Pop App
● Pop - https://marvelapp.com/pop
● Combining sketching and interactivity on mobiles
● Take pictures of sketches, link pictures together

133. Using Pop

134. Pop Demo
https://www.youtube.com/watch?v=Zdjg9yhBqTU

135. Proto.io
• Web based prototyping tool
• Visual drag and drop interface
• Rich transitions
• Scroll, swipe, buttons, etc
• Deploy on device
• mobile, PC, browser
• Ideal for mobile interfaces
• iOS, Android template
• For low and high fidelity prototypes

136. Proto.io Demo
https://www.youtube.com/watch?v=omw
1o0IoSNw

137. Case Study: CityViewAR (2021)

138. February 2011

139. CityViewAR Application
Goal: Create a mobile app that allowed
people to view immersive panoramas on
site, showing the 2011 damage
Key technology: Mobile phone, GPS, 360
panorama, map
User Experience: People walking the city
streets will be able to be immersed in the
earthquake damage from 10 years ago

140. CityViewAR Prototyping Process
1. Sketch Interaction Flow (Paper)
2. Create Wireframe (Powerpoint)
3. Sketch Screens (Paper)
4. Test Transitions (Pop)
5. Collect/Create Assets (Photoshop, various)
6. Create High Fidelity Prototype (Proto.io)
7. Code (Unity, Mapbox)
8. Make video demo (Various)
9. Publish (Android, iOS)

141. Sketch Interaction Flow/Wireframe

142. Interaction Flow

143. Sketch Interface

144. Pop Wireframe

145. Pop Demo
https://www.youtube.com/watch?v=2XTgvjHDKlo

146. Asset Collection

147. High Fidelity - Proto.io

148. Test Transitions
https://youtu.be/eGxgpzXUiMQ

149. Final Interface Design

150. Unity Build

151. Deployment to App Store

152. Demo
https://www.youtube.com/watch?v=4pL0sEB1884

153. SUPPORTING INTERACTIONS

154. AR Visual Programming
• Rapid prototype on desktop
• Deliver on mobile
• Simple interactivity
• Examples
• Zapworks Studio
• https://zap.works/studio/
• Snap Lens Studio
• https://lensstudio.snapchat.com/
• Facebook Spark AR Studio
• https://sparkar.facebook.com/ar-studio/

155. Lens Studio
https://www.youtube.com/watch?v=jOeldwY5eRI

156. Lens Studio
Author and preview AR prototypes
● Tool behind Snapchat Lenses, but
also a powerful AR prototyping tool
● Can do face (using front camera)
and world lenses (rear camera)
● Simulated previews using webcam
Deploy and use advanced AR features
● Can deploy to phone running
Snapchat app via Snapcode
● Has advanced AR tracking and
segmentation capabilities
Some specific features
● SnapML: train and use ML models
● Face landmarks: track 93 points
● Face expressions: detect winks etc.
● Eye tracking: get eye pos/rotation
● Behavior: set triggers & actions
● UI: add UI widgets without scripting
● Visual scripting: no code via graphs
● Templates: lots of templates and
tutorials (we’ll explore later)

157. Example: Museum AR Experience
A. Cardenas (2019): I Used Snapchat’s Lens Studio
to Prototype Augmented Reality Apps and This is
What I Learned
Original art piece by
Penelope Umbrico on
display at UMMA One of our
lenses
On-site observation &
Snapchat access statistics

158. Zapworks Studio
https://www.youtube.com/watch?v=dK6BmOsRxbw

159. Creating On Device
•Adobe Aero
•Create AR on mobile devices
•Touch based interaction and authoring
•Only iOS support for now
•https://www.adobe.com/nz/products/aero.html

160. Aero Demo
•https://www.youtube.com/watch?v=1F83n8W2JUg

161. Apple Reality Composer
• Rapidly create 3D scenes and AR experiences
• Creation on device (iPhone, iPad)
• Drag and drop interface
• Loading 2D/3D content
• Adding simple interactivity
• Anchor content in real world (AR view)
• Planes (vertical, horizontal), faces, images

162. Apple Reality Composer Example
https://www.youtube.com/watch?v=tRcAGdM7878

163. Apple Reality Composer

164. DEVELOPMENT TOOLS

165. Importance of Physical and Digital Prototypes
• Quick & dirty
• Explore interactions
• Get initial user feedback
• Avoid premature commitment
• Devise technical requirements

166. Content and Interaction
Polished
Content
Placeholder
Content
Implicit & Explicit
Interactions
“click”
Lo-Fi Hi-Fi

167. Development & Testing
WebXR
THREE.js AR.js
A-Frame ...
Unity /
Unreal
SteamVR
AR
Foundation
MRTK ...
Native SDKs
Oculus
ARKit/
ARCore
VIVE ...

168. Paths to Being an XR Developer
Web dev
background
Game dev
background
Mobile dev
background
“Easy”
“Hard”
WebXR
THREE.js AR.js
A-Frame ...
Unity /
Unreal
SteamVR
AR
Foundation
MRTK ...
Native
SDKs
Oculus
ARKit/
ARCore
VIVE ...

169. A-Frame
• Based on Three.js and WebGL
• New HTML tags for 3D scenes
• A-Frame Inspector (not editor)
• Asset management (img, video,
audio, & 3D models)
• ECS architecture with many open-
source components
• Cross-platform XR

170. Hello World in A-Frame
https://aframe.io/examples/showcase/helloworld/ https://aframe.io/docs/1.0.0/introduction/

173. Unity
• Started out as game engine
• Has integrated support for many
types of XR apps
• Powerful scene editor
• Asset management & store
• Basically all XR device vendors
provide Unity SDKs

176. Web vs. A-Frame
Structure & Content
HTML
Organization of page content
& hierarchy
Presentation
CSS
Definition of page
content presentation
Behavior
JavaScript
Specification of interactive
behavior
Web
A-Frame
Structure
Entities
Organization of 3D
scene hierarchy
Content & Presentation
Components
Definition of 3D scene
content & presentation
Behavior
Systems & Scripts
Specification of interactive
behavior

177. WebXR & A-Frame
Web
HTML
<div id><!--
… --></div>
CSS
#id { left: x; top:
y; z-index: z }
JS
function f () {
… }
content + structure presentation behavior
WebXR
navigator.xr
.requestSession
WebGL
<canvas class=“a-canvas” />
THREE.js
entity.object3D
A-Frame
<a-entity id position=“x y z”>
</a-entity>

178. WebXR
Enable XR applications on the web by
allowing pages to:
• Detect XR capabilities
• Query XR device capabilities
• Poll XR device and associated input
device state
• Display imagery on XR device at the
appropriate frame rate
Is XR mode supported?
Advertise XR mode to
user
AR VR
Enter XR mode is
triggered?
Request XR session
Render frame loop in
session
Exit XR mode is
triggered?
End XR session
↻

179. Unity vs. A-Frame
Unity is a game engine and XR dev
platform
● De facto standard for XR apps
● Increasingly built-in support
● Most “XR people” will ask you about
your Unity skills :-)
Support for all XR devices
● Basically all AR and VR device
vendors provide Unity SDKs
A-Frame is a declarative WebXR
framework
● Emerging XR app development
framework on top of THREE.js
● Good for novice XR designers with
web dev background
Support for most XR devices
● Full WebXR support in Firefox,
Chrome, & Oculus Browser

180. XR Platforms & Toolkits
AR
VR
A-Frame
SteamVR
MRTK
XR Interaction
Oculus
VIVE
Marker-based
AR.js
Vuforia
Marker-less
A-Frame
ARKit/ARCore
AR Foundation
MRTK
XR Interaction

181. XR Platforms
WebXR

182. XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

183. XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

185. XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

187. XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

189. XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

191. XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

193. XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

195. XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

197. RESOURCES

198. XR Prototyping Web Site
XR Prototyping resources (http://xr-prototyping.org/)

199. XR MOOC
XR MOOC on Coursera (http://xrmooc.com) -- free audit possible

200. XR MOOC Office Hours YouTube Playlist
https://www.youtube.com/playlist?list=PL2rpGQebh6lAVtGBIUaUz2yGbZCdqrEed

201. Sketching Resources

202. Other Books
3D User Interfaces: Theory and Practice
- Joseph J. LaViola Jr.Ernst Kruijff, Ryan P.
McMahan, Doug Bowman, Ivan Poupyrev
Augmented Reality: Principles and Practice
- Dieter Schmalstieg, Tobias Hollerer
Creating Augmented and Virtual Realities: Theory
and Practice for Next-Generation Spatial
Computing
- Erin Pangilinan, Steve Lukas, Vasanth Mohan

203. Tools and Guidelines
The UX of VR
Low-fi prototyping tools for VR
3D design patterns for VR Virtual Reality Human Interface Guidelines

204. Useful Links
VR paper prototyping
https://blog.prototypr.io/vr-paper-prototyping-9e1cab6a75f3
Draw sketches for VR
https://medium.com/inborn-experience/vr-sketches-56599f99b357
Wireframing 3D UI
https://arvrjourney.com/lesson-3-4-story-flow-and-wireframe-3d-interactions-dc3c7d5cb79a

205. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au