GDC 2016 Bringing Real World into Game World with RealSense

GDC 2016
Bringing the Real World into the
Game World with RealSense Technology
Geoffrey Douglas, Games Engineer
@geofdouglas

Intel Software – Developer Relations Division Intel Confidential
About Me

Setting The Tone
RealSense
enables fun
gameplay
experiences
Intel knows
technology
But YOU
know games
People love
new stuff

Acknowledgments
Trapper McFerron
Joe Scheinberg
Kelly Gawne
Yaz Khabiri
Raja Bala Doug McNabb
Oscar Nebres
Jeff Williams

Agenda
5
RealSense Technology Overview
Gaming Use Cases
Personalization
Socialization
Conclusion / Next Steps
Cameras, SDK Capabilities
Future Plans, Getting
Involved
Q & A
BGS, 3D Scanning,
Augmented Reality

Intel Software – Developer Relations Division Intel Confidential 6
RealSense – Depth Imaging Technology

Coded Infrared Light

User-Facing
Short-Range (6” – 4’)
One IR Sensor
Windows 10
Laptops and Desktops
World-Facing
Long-Range (18” – 10’)
Two IR Sensors
Windows and Android
Tablets and 2-in-1s
RealSense Cameras

RealSense SDK
SDK
Core I/O
Modules
Algorithm
Modules
C++ Interface
Algorithm
Modules
C# Interface Unity* Interface JavaScriptJava Interface
Unreal Engine 4
Interface

RealSense SDK Middleware
Hand Tracking
Speech Recognition
Blob Tracking
Head/Face Tracking
Gesture Recognition Scene Perception
Background
Segmentation
3D Scanning

RealSense SDK Middleware
Blob Tracking
Low-precision
object tracking
Designed for
hands and faces
Segments “blobs” and
draw contour lines
Hand Tracking
22-Point
Feature Tracking
Tracks both hands
Zero-Latency
“Cursor” Mode
Gestures
Thumbs Up / Down
V-Sign
Spread Fingers
Fist
Swipe
Wave
Tap
Full Pinch
Two-Finger Pinch
Face Tracking
72 Facial Landmarks
Head Pose
Expression Detection
Pulse Rate Detection
Background
Segmentation
Separate User
From Background
3D Scanning
Face / Head
Objects
Full Body
Textured / Vertex Colors
Solidification

How does this all apply to video games?

Well first…why do people play video games?
Engaging Storytelling
Escape
Wish Fulfillment
Socialize Exploration
Adventure
Relaxation
Creation
Interactive Artform
FUN!!!
Achievement
Puzzles
Competition
Sharing

So… where does RealSense fit in today?
Escape
Wish Fulfillment
Adventure
Relaxation
Creation
Interactive Artform
FUN!!!
Achievement
Puzzles
Competition
Sharing

Where will this presentation focus?
Escape
Wish Fulfillment
Adventure
Relaxation
Creation
Interactive Artform
FUN!!!
Achievement
Puzzles
Competition
Sharing
Game Personalization Game Socialization
FUN!!!

Game Personalization

Character Personalization

Asset Generation

Level / World Generation

Level / World Generation (LEGO)

3D Scanning with the RealSense SDK
PXCSenseManager *sm = PXCSenseManager::CreateInstance();
sm->Enable3DScan();
PXC3DScan* scanner = sm->Query3DScan();
sm->Init();

PXC3DScan::Configuration config = {};
config.mode = PXC3DScan::ScanningMode::OBJECT;
config.options = PXC3DScan::ReconstructionOption::TEXTURE;
config.startScan = true;
scanner->SetConfiguration(config);

while(sm->AcquireFrame() == PXC_STATUS_NO_ERROR)
{
PXCImage* scanImage = scanner->AcquirePreviewImage();
...
sm->ReleaseFrame();
}
scanner->Reconstruct(PXC3DScan::FileFormat::OBJ, “scan.obj”);

Game Personalization
Take ownership of your experience through custom content creation
• Character Personalization via Face / Head / Body Scanning
• Asset Generation via Object Scanning
• Level / World Generation via Environment Scanning

Game Socialization

Chat Heads
Level 0
Level 1
Level 2
Level 3
No Comms
Text Chat
Voice Chat
Video Chat

Segmented Streaming

BGS with the RealSense SDK
PXCSenseManager *sm = PXCSenseManager::CreateInstance();
sm->Enable3DSeg();
PXC3DSeg* seg = sm->Query3DSeg();

sm->EnableStream(STREAM_TYPE_COLOR, 1920, 1080, 30.0f);
sm->EnableStream(STREAM_TYPE_DEPTH, 640, 480, 30.0f);
seg->SetFrameSkipInterval(0);
sm->Init();

while(sm->AcquireFrame() == PXC_STATUS_NO_ERROR)
{
PXCImage* segImage = seg->AcquireSegmentedImage();
...
sm->ReleaseFrame();
}

Game Socialization
Enhance your social experience of playing video games.
Depth-Based Foreground / Background Segmentation
• See your teammates and/or competitors “floating heads” while you play together.
• Achieve green-screen style streaming without the large and expensive setup.

Next Steps for Intel
Refine the
implementatio
n and the
performance
of the SDK
Develop more
code samples
demonstrating
these use
cases
Integrate the
features of
the SDK
seamlessly
into popular
engines
Test these
experiences
with gamers to
perfect the
user
experience

Conclusion
RealSense Technology
can be used in many
different ways to
create fun, new gaming
experiences. Players are primed and
ready for something
new, and RealSense
can be a differentiator
for your next game.
Developing with a new
technology like
RealSense is a huge
challenge, but we can
overcome it by
working together.

Getting Involved
Get in touch with us! If you want to talk about adding RealSense capabilities to
your game or if you want more information, we would love to hear from you.
Order a RealSense Developer Kit and start playing with the SDK:
https://www.software.intel.com/realsense
Get excited about the future of gaming! It’s going to be great!
36

Q & A
https://software.intel.com/en-us/articles/chat-heads-and-intel-realsense-sdk-
background-segmentation-boosts-esport-experience
https://software.intel.com/en-us/articles/
applying-intel-realsense-sdk-face-scans-to-a-3d-mesh
https://software.intel.com/en-us/articles/
intel-realsense-sdk-plug-in-for-unreal-engine-4
geoffrey.douglas@intel.com
@geofdouglas
37

GDC 2016 Bringing Real World into Game World with RealSense

GDC 2016 Bringing Real World into Game World with RealSense

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to GDC 2016 Bringing Real World into Game World with RealSense

Similar to GDC 2016 Bringing Real World into Game World with RealSense (20)

More from Intel® Software

More from Intel® Software (20)

Recently uploaded

Recently uploaded (20)

GDC 2016 Bringing Real World into Game World with RealSense