Presented at FITC Toronto 2019 More info at www.fitc.ca/toronto Bushra Mahmood Unity Technologies Overview In this talk, Bushra Mahmood will explain how to articulate and pitch augmented reality as a viable medium to help solve problems. Learn about what makes an AR application come together on both mobile devices and headsets. Uncover different tools and methodologies for problem-solving and making a compelling story. By properly understanding this technology and its parts, creatives can take an active role in shaping and defining this new space in computing. Objective Learn the tools and techniques required to pitch an augmented reality project. Target Audience Designers, product managers, product stakeholders. Assumed Audience Knowledge An understanding of product design and an awareness of AR Five Things Audience Members Will Learn The right language to use when explaining ‘spatial’ design The different requirements and considerations for scoping an AR project The tools that are currently available for AR authoring Insights into what the near and far future will hold for this medium. An example of an AR application pitch

1. How to Pitch
Your First AR
Project
@goatsandbacon
Hello friends, Thank you so much for having me.
I’m really excited to be here and to talk abut how you can pitch your first Augmented Reality project.
What I want to do is share examples and processes that can help you scope and conceive an AR experience. A show of hands people here who have made or shipped an AR project? Okay great, lets share notes after this talk.

2. WHAT IS IT THOUGH?
So before I get started, I want to address something.
There’s a lot of debate around what this industry that we are in called? What field does AR fall under?
To be honest, at this moment there’s no right or wrong answer. I think language is still evolving and once we start seeing more devices and adoption, we’ll have a better idea of what the best thing to unify on is. In that spirit, there may
be language I use in this talk that might change too.

3. Spatial Computing
INTRODUCTION
For the moment Spatial Computing seems to feel the most snug so that’s what I’m going with. We think spatially and space is where we as humans have evolved. Our work involves using computers to extend our presence and
capabilities in spaces around us so… spatial computing just works. This format is also inclusive of VR which is why I like it.

4. Exploring the
horizons of
technology
requires courage
because research
carries risk”
- IVAN SUTHERLAND
http://cseweb.ucsd.edu/~wgg/smli_ps-1.pdf
“
So Ivan Sutherland who is the father of computer graphics, he’s the reason all of us have jobs. And he wrote this phenomenal essay on Technology and Courage which has become my foundation.
The reality about designing in this or any new format really, Is that there are risks involved that you need to mentally prepare for. Technology is moving at an incredibly fast pace and there are a lot of unknowns.
“The risks that are difficult to quantify, that are different than money and resources, they are social and emotional risks, risks to reputation and to pride; they are risks that are felt but difficult to identify and describe.” Basically if you
want innovate, If you want to truly innovate and be one step ahead, theres no room for pride or ego.

5. “My boss wants me to
make an AR thing…
What should I make?”
That being said, there’s are some pretty OBVIOUS risks you can probably avoid.
This is a sentiment I’ve ran across several times over the past few years.
There is no ‘AR Thing’. There are problems that need solving, and these problems have constraints that may be overcome by thinking spatially. The first step in finding out if AR is the right medium is by identifying the users and their
needs.Do these problems involve immersing the user in real time, assisting them in space or physically engaging them? Are there any physical constraints that currently prevent them from being successful? If so, then there’s a good
chance that Augmented Reality can add value to the solution.
Having your users experience rather than observe may sharply increase the chances of them understanding and retaining information.

6. Understanding
Mobile AR
BREAKDOWN
For this talk I will specifically focus on Mobile AR - the reason for this is that most if not all of your audience will have a phone. This is the most scalable medium that’s in circulation right now. By getting ourselves familiar with this
medium first, we become primed for head mounted displays and any other device that will come in the future. However, a mobile phone was not necessarily designed for AR experiences and therefore has a lot of points of friction.

7. Front Facing (Notch)
World Facing Camera
Machine Learning Capability
Front
Back
A. Infrared Camera (Reads Data)
B. Floor Illuminator (Face/Eyes Detection)
C. Proximity Sensor (Activity/Human Detection)
D. Ambient Light Sensor
E. Speaker
F. Microphone
G. True Depth 7MP Camera
H. Dot Projector (Face Recognition)
Dual Lenses
Image Stability
Neural Engine
Surface/Image Detection
GPU (Realtime 3D Rendering)
A B
C
D E F G H
iPHONE XS
But we really should understand how the phone works beyond just the operating system or interface.
Hardware is easy to overlook when focusing on interface design and experience. It’s also not something we are taught about in design school.
But if you’re working in new and emerging tech, knowing how things work can help inspire and even bring innovation to this space. Just a general awareness is enough to get some inspiration
Hardware is also generally easier to scope and predict than user behaviors. It is vital for designers to explore beyond current technological constraints so they can help lead the technology forward.

8. Depth Sensor Gyroscope MagnetoMeter Accelerometer Light Sensor
EXAMPLES OF SENSORS
The most common method of ingesting information is through a camera. However, the most accurate way is to use sensors. A sensor is a piece of technology that detects information from its surroundings and then responds back
with data. Really consider how a sensor can enhance your users experience.
Depth Sensor: Calculates depth and distance.
Gyroscope: Detects the angle and position of your phone.
MagnetoMeter: Essentially a compass that can always tell where north is.
Accelerometer: Detects change in velocity, movement, and rotation.
Light Sensor: Measures light intensity and brightness.
Having these in your toolkit will really help you consider more possibilities

9. EXAMPLES OF MACHINE LEARNING
Facial
Recognition
Visual/Object
Recognition
Personalization/
Suggestions
Predictions Chatbots
It’s one thing to gather data but another to make it meaningful. For example, calculating the distance between a phone and a door can be easy with a sensor. However, trying to identify the door can be extremely difficult. The
computer must understand what makes a door different from a wall or a window? It must also comprehend the different types of doors that it is likely to encounter.
Facial Recognition is a super common ML found in camera apps and face filters.
Visual and Object recognition identifies not just things around us but visual cues such as lighting conditions
Personalization is probably the most common ML type you encounter day to day, it’s how your profile is shaped on social media
Predictions is also very common, especially in apps like Lyft and Uber where the app gives you an estimated time of arrival
Chatbots is essentially conversational UI that can learn through repetition and become more customized for unique cases, super common for support roles.

10. Defining Purpose &
Value
RESEARCH
Properly researching and defining the problem space you are building for helps you in several ways. It provides you some evidence and data to base your assumptions off of. It helps you narrow your possibilities and avoid remaking
something that already exists, It also might also make you rethink your hypothesis and if its the right one. Most of all it really connects you to the user you are designing for.

11. DEFINE USER BEHAVIOR
Interact with
Content
IKEA Place
Follow a Narrative
Ghostbusters
World
Capture Data
Tap Measure
A really easy place to start is to figure out what the focus of your users behavior will be - is success based on them just interacting with something, like in Ikea place, where they view and place furniture to decide that yeah, I would
buy this. Then you should scope for an interactive experience where the user will be able to manipulate content and have some form of control.
In a narrative first experience, the success of the experience relies on the user performing a sequence of actions in order to reach a final goal or destination.
Narrative first experiences are most often used for gaming and story based experiences, they also rely a lot on device orientation, location and maps. Or if Ghost bustin’ makes me feel good.
In a capture first experience, the success of the experience relies on the user ultimately capturing content like a video or an image. A capture first experience is the basis of most sharing and platform based applications. It involves
getting the user to create an artifact they can then take with them out of the app or in a different mode.

12. ENTERPRISE / EDUCATION
Learn /
Train
iNaturalist
Preview/ Try
Before you Buy
WannaKicks
Instruct / Assemble
Golfscope
After you’ve figured out what your user is expected to do, ask yourself, what category will this fall into?
These are all examples of real applications in the market currently in the App store.
There is tons of metrics and good data on these use cases that you can use to really sell your vision.
I’d say we are at a moment where it can be possible to do a viable competitive analysis or audit. You can really zoom in on one of these several categories and start from there.
An example is iNaturalist that lets you learn about wildlife around you
Preview/Try Before you buy like Wannakicks that lets you try on sneakers before you spend $1000 bucks
There’s also instruct or assemble apps like Golfscope that help you improve your golf game.

13. CONSUMER
Social
Facebook
Messenger
Gaming
Pokemon GO
Brand
Engagement
Miller Lite by
Trigger
Facebook Messenger offers a ton of social interactions where you can use the front facing camera to play with friends.
Pokemon GO - This is a great example of why research is important, The fact is that most players in Pokemon GO actually disable the camera AR feature when catching a Pokemon because it actually makes the game harder -
however there’s TONS of other AR capabilities that are worth looking at such as the actual weather influencing the Pokemon you see, or location based data.
Then we have brand engagement like this miller lite campaign made by Trigger for St Patricks day where a little friend pops up and serenades you as you drink.

14. UTILITY/ TOOLS
Measurement/
Accuracy
KLM Airlines
Location/Guide
Google Maps
Creative /
Authoring
Facebook Stories
KLM Airlines lets a user double check to make sure their carry on fits in the overhead compartment.,
Google maps is great for people like me who will forever struggle with left and right, also which way Missisauga is.
Facebook Stories activating a real-time drawing system where you can design digital paintings and then pose them throughout the real-world.

15. ART & DELIGHT
Face Meme
Dan Miller
Play the City
Tool of North
America
Childish Gambino Playmoji
Google
Playground
The there are Apps that are just fun and bring delight. These types of applications and experiences also tend to be the most experimental and really push the medium to
its potential.

16. Language &
Foundation
CONTENT
So after you’ve determined what problem space you are looking to work in.
Paying attention to the language you use and how you describe things is important on so many levels, words have meaning and if you’re all on the same page it creates
alignment with you and the rest of the stakeholders. You’re sort of in uncharted waters so make the effort to be on the same page. One less confused person goes a long
way in your product cycle.

17. So The entire point of AR graphics is to overlay data in space, That means it’s a 3D experience even if you’re working with 2D graphics. If your experience isn’t visual, it’s still happening in real time.
Traditionally, in 3D software you would craft your scene and then to properly view it you would render it out after you make something.
AR as a medium cannot afford any lag or delay so it needs to happen in real time.
Get yourself comfortable with the term Real Time 3D because it’s a known authoring convention that has a ton of process and documentation that can help inform you and make smarter decisions.

18. Static
Content that is still
and lacks movement
and interaction
Animated
Content that moves on
a timeline or follows a
sequence
3D
Object with width,
height and depth.
Dynamic
Adaptive content that
changes with
interaction or over time
Procedural
Content generated
automatically or
algorithmically
CONTENT TYPES
The following are examples of some of the more popular content types used in spatial computing.
These content types are not exclusive and can combine in many different ways. However, it is essential to understand these formats so the designer can properly articulate what they are trying to do.

19. Dynamic 2D Flexible Glass Dynamic 3D Fixed Space
DESCRIBING BEHAVIOR
Type State Location State LocationType
When mapping out behaviors and relationships, it is helpful to be specific about where and how to treat the content. Try to be as precise in describing the experience to get alignment amongst stakeholders.
A good rule of thumb is to call out the type of content, the state and the location.

20. ADDITIONAL RESOURCES
Designers Guide to AR
https://medium.com/@goatsandbacon
3D For Designers
www.3dfordesigners.com
I’ve actually written quite extensively about content types which you can read on Medium. If you’re an absolute beginner to 3D, I would recommend 3D for Designers, It was Created and taught by design industry veteran Devon Ko, it
includes both free lessons and a comprehensive 3D design fundamentals course. I’m also a TA on this course so come say hi!

21. Discover, Prototype
& Design
TOOLS
Once you’ve figured out what type of app you’re going to make, the next step is to figure out to how your app will look and what features you’d like to invest in.

22. DISCOVER: GAMES
https://80.lv/articles/the-design-secrets-of-breath-of-the-wild/
Zelda: Breathe of
the Wild
Nintendo
Overwatch
Blizzard
Conduct AR
Northplay
In the discovery process you will do yourself a great disservice if you don’t look at 3D games.
3D games are probably the most similar in design treatment and challenges since in both 3d games and AR. your layering real time 3D data over a camera controlled by the user.
There’s a lot of visual treatments done in gaming that can inspire your interfaces - for example overwatch uses color burns to make sure their UI works on both light and dark surfaces
Another reason to look at games is for the game theory and tricks they use to make a more compelling game play - Matt Walker who is Production Manager for Capcom translated a series of tweets by the game director and senior
lead artist for Zelda Breathe of the Wild.
He talks about the “Triangle Rule”. By using triangle shaped obstacles they complete 2 objectives- gives players a choice as to whether to go straight over the triangle, or around, and it obscures the player’s view, so designers can
utilize them to surprise players, make them wonder what they’ll find on the other side.

23. RAPID PROTOTYPE
Storyboard
Invest in a clear story from beginning
to end to determine a feature.
3D Room Simulations
Use 360 videos, 3D models or a
sequence of snapshots to simulate an
environment
Animated Clickthrus
Use timed transitions using Framer or
Adobe XD to create clickthrus on your
phone to get a sense of ergonomics
and interactivity.
One of the challenges in making static prototypes is that things are always changing in your camera feed, and so using methods that have animation or follow a timeline
and narrative can help communicate the changing nature of AR and your interface.

24. TRADITIONAL COMPOSITING
After Effects
Import phone footage and use
3D Camera tracker and
camera solve to create 2.5D
simulation
Cinema 4D
Import image sequence into
Motion tracker to generate 3D
camera information
If you can composite or do some motion graphics its even easier to conceptualize how UI might look on a moving camera, If you’re not as familiar, there are tons of tutorials online to help you do motion tracking. Tracking software is
super helpful. Tracking basically means to overlay position size and rotation data of something over a video so that it looks like it belongs in the same scene. A lot of principles of making things look like they belong in video
compositing can be inherited in AR.

25. DISCOVER : PLAY
You have to play to discover opportunity. I find it super helpful to challenge myself with random pieces of footage that I find around me. It’s a super low pressure way to
just create and learn.

26. REAL AR AUTHORING
AR Foundations (Unity)
iOS & Android
Project Aero (Adobe)
Currently in Beta
Torch AR
iOS
There’s really nothing quite like the real thing - There are a few options out there to prototype and author experiences. There are tons of tutorials and information online on
how to get started with AR foundations in Unity - The upside to mocking up in Unity is that your entire app that you plan on publishing can be made once in Unity and be
built out to ARKit and ARcore.

27. Limitations of
Mobile AR
HEADSUP
So we’ve discussed figuring out what your user wants to do, what category you want to focus on, how you can start prototyping and getting a feel for your idea. But there are a few areas of friction that I’ve personally encountered that
you are likely to encounter too.

28. Semantic
Understanding
Visual
Inconsistency
Platform File Extensions Relocalization
CURRENT HURDLES
?
Triangle
The promise of AR is its integration into the real world, yet we’ve seen very limited examples of that. Right now it’s tricky to get the world to interact back with you. These are a few of the hurdles I’ve faced.
Semantic understanding basically means does the device know exactly what something is, how it should behave? This technology is not mature enough to be reliable for anything that isn’t a basic model. For example ARkit and
ArCore don’t tell you what the floor is.
Visual Inconsistency : Things just feel like they float and don’t quite look right, things don’t occlude or match the expected perspective.
Platform: Some things that are specific to a device or platform and not really in your hands or control - take the time to learn about these and if you can, the company roadmaps.
File Extensions: the industry has divided itself in terms of support. For example, Apple exclusively supports and encourages the use of .USDZ for ARKit applications, however Chrome does not support this format on the web and
promotes .glTF. If you are considering commissioning or buying a 3D asset, my recommendation would be an .fbx since you can covert it into both .glTF and .USDZ based on your needs.
(Relocalization is where the device knows its position and orientation in space, that helps create a persistent experience that then allows for returning and multiplay experience - so it means things are where you had left them or
someone else can visit an experience you were just in in the exact space

29. Architects of a New
Reality
LEVEL UP
I want to challenge you to think beyond what the current apps that I’ve shared are.
The amount of information and access to information we will have will grow and so how can you prepare yourself to create the next generation of apps? This is a phrase
I’ve used before in Context design and it makes a lot of sense to consider yourself an architect since you essentially design spaces

30. This is Ellen, a 3D model that you can download for free from the Unity asset store under the 3D game kit.

31. CONTEXT OF A TABLE
Elevated from Floor Small Table Size Horizontal Plane
Lets say you want to have your 3D model appear on a small table.
So what is a table context?
Context is basically the circumstances that form a setting for an event or idea. If the following requirements are met, I want Ellen to spawn on the table.

32. The reality is that you have no control over what a users room could look like, at all. This is basically the fuzzy area you have to design for.

33. But if you can design different bits of contexts and figure out a way to make your experience work in an unknown environment

34. You can figure out how to make it work in any environment - and really extend the value of your design. This is true immersion because you adapt contextually and
responsively to the space around the user.

35. Mixed Augmented
Reality Studio
(Project MARS)
* Currently in Preview
UNITY LABS PRESENTS
This is a great way to talk about what our team is working on in Unity Labs.

36. CONTEXT FIRST AUTHORING
Context
Content
What would otherwise take a lot of time and math we are making into easy to use visual tools so you have less mental overhead when embracing this new type of
authoring.

37. PROJECT MARS
This is what the app allows you today. In MARS you basically have objects that you give affordances to, and then in the app you can preview & iterate against simulated
& real data.
We give you a set of rooms that you design against and can customize. This makes you create as many contexts as you need to match as many spaces with the desired
experience.

38. PROJECT MARS
Here is something that was built in MARS today - What’s great is that as a user starts scanning, you can also design an experience that adapts and changes as they get
more of a room and different surfaces. This sort of procedural authoring could change AR from being these bite sized experiences to something you actually live and
interact in. If you’re interested in trying it out for yourself, definitely talk to me after. I’ll also have my email up at the end.

39. When human
systems and tool
systems align - it
leads to an
accelerating rate of
progress”
- DOUGLAS ENGELBART
GROUP OF
EXPERTS
HARDWARE/
SOFTWARE
DATA/
INFORMATION/
KNOWLEDGE
COLLECTIVE INTELLIGENCE
“
I kinda want to end this talk on a note where I need your help for very selfish reasons - the more we can help to improve our collective intelligence, the better it is for me..
I mean us.
For real though, I am deeply interested in how humans and tools evolve, I strongly believe that how we all work together on important challenges is a measure of our
progress and how effective we as a society are.
I think the pace in which tools evolve can be really good indicators of progress.
DOUGLAS ENGELBART

40. Thank You!
bushram@unity3d.com
@goatsandbacon