Hello, My name is Daiki Taniguchi from mobile game development corporation called Akatsuki.
This session is “A[ou]ugmented Reality Game
with Unique Semi-Transmissive Rendering Method&quot;.
OK, Let&apos;s start.
At first, let me show our product.
This is our AR based first person shooter.
Today I gonna talk about How can we archive immersive AR experience?Through development of this game, I got a hint of an answer to this question and I’d like to share it.
First, let&apos;s think about the key elements of AR experience.
We think, there are 3 key elements in AR.
First, geometric consistency. It is related to tracking physical world.
Next, optical consistency. It is about blending Computer Graphics into physical world with no incompatibility.
Last, time consistency. It can be rephrase fulfilling target FPS. But there is no need to focus this theme because it could be included in every elements.
In this time, I’d like to focus on optical consistency is strongly related to computer graphics and we must be interested in.
Now, we could start with next question.
How can we smoothly integrate 3D Computer Graphics into physical world?
To keep great optical consistency, there are 2 simple rendering approaches.
The one is Photorealistic Rendering, the other is Non-photorealistic Rendering.
Photorealistic Rendering is so straight forward way because high photorealistic object must be blended into physical world smoothly.
But this approach has some difficulties. Because, at the moment, we could have only AR devices which has very limited CPU or GPU resource. But, In photorealistic rendering we must do many complicated operations like light estimation, calculating BRDF, casting shadow etc. When we try to achieve sophisticated immersive experience, it will cause critical performance issues.
Of course we have continuing research about photorealistic approach too, but in this time, we selected the other. It is Non-photorealistic Rendering, also known as NPR
These are typical NPR examples.
NPR can be desirable way to unify image style between 3D objects and environment.
But It has a problem in terms of Reality. Of course, it&apos;s obvious, because NPR means &apos;Non-photorealistic&apos;
Can we achieve AR optimized NPR?
And if we can, how is that?
The key is &quot;Story backed experience”.
This is our coined word and “story” means famous movie, novel or game like this.
So &quot;Story backed experience” means that making player evoke feeling as ‘I&apos;ve seen this scene before! In movie, novel, game …’
How can we achieve this? Now I show you some videos.
How is this? Have you ever seen this in a movie?
Yes, this is DeathStar! You could remember R2D2 is projecting hologram.
And do you know this too?
Someone could remind Pre[:]dator. Camouflaged predator attacking scene was so thrilling.
And, more recent one, Do you know Ghost in the shell? It was born in my country, Japan.
Hey, I guess everyone should know this. It&apos;s T-1000 in terminator2. It was so frightening.
Now how do you think?
These objects should not exist in this physical world certainly, but you could feel these as real, don&apos;t you?
So far I’ve explained a concept of AR optimized NPR, now I gonna explain how to applied this method to our game.
This is semi-transmissive enemies, they are fit the physical world perfectly. How could we reach to this? I show the process.
First, The[zi] advantage of non-photorealistic rendering is
- it could omit complicated procedure like photorealistic rendering.
From this, we can pull out an suggestion.
- A material should be organized without shadowing, lighting and occlusion.
Second advantage of NPR is
it could make border ambiguous between physical world and computer graphics.
And we pull out next suggestion.
- Physical world or background image also could be regarded as processing objects.
From these suggestions, technique related to transmissive or composite background image can be nominated. And we decided to adopt projective texture mapping.
Projective texture mapping is popular technique for graphics programmer, but in AR I don’t know example of utilization of this. So I gonna explain it simply.
In this technique, a texture image to be projected onto the scene as if by a slide projector.
- UV values should be calculated dynamically unlike standard texture mapping
- Transformation matrix is required to bridge a gap between projective coordinates and texture coordinates
so, this technique go very well with AR, like this, projecting background image to enemy models in real-time.
Of course we research and consider about other techniques related to AR optimized NPR like machine learning based one and so on, but this time we use this simple technique.
Next, holographic shader is based on simple texture mapping.
- Increase arbitrary RGB color with additive
- Disable RGB values exceeding a threshold
- Add vertex animation with noise
And in fig1, we use two devices for tracking position and rendering. Tracked position is sent to rendering one in realtime.
In fig2, Tracked position by Leapmotion is sent, too.
Last is edge detection.
Semi-transmissive shader is replaced to diffuse shader only within range using stencil buffer, and sobel filter is applied as post effect.
These are so simple algorithm, are they?
And NPR procedure could be much cheaper than Photorealistic-rendering.
In fact our this game achieved 60[sixty] fps.
When we aim at “Immersive AR experience with current generation device&quot;, we believe NPR is so desirable choice.
I showed how to improve optical consistency so far.
Besides it, we should also treat other than graphics elements in order to improve immersion further more.
And it could be how-to interact to the AR world.
Moreover, how to make the best use of semi-transmissive rendering?
It is straight forward way that making this game intensive battle one with optical camouflage.
But please wait. Can we fight against enemies firing a bunch of bullets or magic spells in this physical laws? It might be difficult unless you are superman.
This time we focused on fear from enemy&apos;s semi-invisibility. We introduced spatial audio, we try to evoke an experience like this
- You look around and feel ‘Is there anything?’
- Turn to there but noting exist
- But look carefully, can you see it...?
- You must feel something are everywhere.
In this way, you should find enemies carefully and immerse into AR world naturally.
At last, Conclusion.
Well designed NPR is very promising method when we aim to achieve great AR experience with current generation AR devices.
Story backed experience is key for maximizing effectiveness of NPR.
To Reinforce[Ri inforce] game experience, we should take care of not only graphics but also elements to interact the AR world like physical content, spatial audio and so on.