The document discusses the Gear VR Framework (GVRf), an open source Java library for developing VR applications for Android on the Gear VR. It provides an overview of GVRf's features like loading 3D models and textures, scene management, and handling head tracking. It demonstrates how to create a basic "hello world" app and discusses new features in development like standard shapes, scene objects for video/camera/webviews, scripting support, and improved 3D modeling capabilities. A key focus is the new 3D Cursor interaction model and support for various VR input devices through a common API to standardize interaction across apps. Live demos of hand/eye tracking were shown to illustrate cursor behaviors.
A talk from the Develop Track at AWE USA 2017 - the largest conference for AR+VR in Santa Clara, California May 31- June 2, 2017.
Philipp Nagele (Wikitude): What's Next with Wikitude
An in-depth look into the recent developments at Wikitude and what the next version of the Wikitude SDK will offer augmented reality developers.
http://AugmentedWorldExpo.com
Learn how to use the Lightweight Render Pipeline to optimize for maximum performance on mobile platforms. After watching this, you will be ready to build an experience for standalone mobile VR headsets, like Oculus Go, and how to use 3-DOF controllers for interactions.
Speakers:
Dan Miller (Unity Technologies)
Persistent world-scale AR experiences with ARCore Cloud Anchors and AR Founda...Unity Technologies
In this session, Google engineers will walk through ARCore's Cloud Anchors feature, which will soon allow world-scale persistent augmented reality experiences. Cloud Anchors gives developers the ability to create AR experiences that are shared by multiple users both simultaneously and across a large time frame. With the new ARCore Extensions for AR Foundation, it is easier than ever to create shared AR experiences between users, regardless of what mobile device they are using. This presentation will include a special preview from our partners, Sybo and iDreamSky, about their new experience using persistent Cloud Anchors.
New 2D World-Building, Animation & Graphics Features in UnityUnity Technologies
This video explores organic world-building with Sprite Shape as well as improvements to Tilemap for grid-based layouts. This includes a demo of Phase 2 of our 2D Sprite Rigging tools for 2D animation and enhancements to 2D graphics, including the Pixel Perfect Camera. There's also a sneak peek into the future of 2D at Unity.
Speakers:
Alessia Nigretti (Unity Technologies)
Rus Scammell (Unity Technologies)
This talk was given at CESEC 2015 which is a Summer School on Critical Embedded Systems: http://cesec2015.sciencesconf.org/
These slides present the Arduino Designer why and how we develop it with Eclipse Sirius:
http://www.eclipse.org/sirius/
It presents also the Eclipse Sirius Animation feature:
http://www.eclipse.org/sirius/lab.html
A talk from the Develop Track at AWE USA 2017 - the largest conference for AR+VR in Santa Clara, California May 31- June 2, 2017.
Philipp Nagele (Wikitude): What's Next with Wikitude
An in-depth look into the recent developments at Wikitude and what the next version of the Wikitude SDK will offer augmented reality developers.
http://AugmentedWorldExpo.com
Learn how to use the Lightweight Render Pipeline to optimize for maximum performance on mobile platforms. After watching this, you will be ready to build an experience for standalone mobile VR headsets, like Oculus Go, and how to use 3-DOF controllers for interactions.
Speakers:
Dan Miller (Unity Technologies)
Persistent world-scale AR experiences with ARCore Cloud Anchors and AR Founda...Unity Technologies
In this session, Google engineers will walk through ARCore's Cloud Anchors feature, which will soon allow world-scale persistent augmented reality experiences. Cloud Anchors gives developers the ability to create AR experiences that are shared by multiple users both simultaneously and across a large time frame. With the new ARCore Extensions for AR Foundation, it is easier than ever to create shared AR experiences between users, regardless of what mobile device they are using. This presentation will include a special preview from our partners, Sybo and iDreamSky, about their new experience using persistent Cloud Anchors.
New 2D World-Building, Animation & Graphics Features in UnityUnity Technologies
This video explores organic world-building with Sprite Shape as well as improvements to Tilemap for grid-based layouts. This includes a demo of Phase 2 of our 2D Sprite Rigging tools for 2D animation and enhancements to 2D graphics, including the Pixel Perfect Camera. There's also a sneak peek into the future of 2D at Unity.
Speakers:
Alessia Nigretti (Unity Technologies)
Rus Scammell (Unity Technologies)
This talk was given at CESEC 2015 which is a Summer School on Critical Embedded Systems: http://cesec2015.sciencesconf.org/
These slides present the Arduino Designer why and how we develop it with Eclipse Sirius:
http://www.eclipse.org/sirius/
It presents also the Eclipse Sirius Animation feature:
http://www.eclipse.org/sirius/lab.html
Philipp Nagele (Wikitude): Context Is for Kings: Putting Context in the Hands...AugmentedWorldExpo
A talk from the Develop Track at AWE USA 2018 - the World's #1 XR Conference & Expo in Santa Clara, California May 30- June 1, 2018.
Philipp Nagele (Wikitude): Context Is for Kings: Putting Context in the Hands of AR Developers
In this session, Philipp Nagele will explore why AR centers all around context and why contextual understanding is fundamental to any AR experience. He will show how Wikitude is trying to solve this problem for AR developers and provide technical details about the new release of the Wikitude SDK.
http://AugmentedWorldExpo.com
Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016Codemotion
This year for the Salone del Mobile at frog, we came up with a funky experiment, based on Drones, Android, and VR. In this talk, your hosts will walk you through our Drone Race experiment, touching topics like real-time computer vision, reactive programming for mobile, indoor positioning and (wheeled) Drones hacking. The variety and complexity of these topics is equal to its coolness though, so you may be puzzled asking yourself: "Where do I start?" We will share experiences and lots of code, so that you can start right away.
Get to know the Wikitude SDK in its details. Learn what the Wikitude SDK is offering and how it is structured. See the advantages of the Native API vs. the JavaScript API.
Augmented World Expo (AWE) is back for its seventh year in our largest conference and expo featuring technologies giving us superpowers: augmented reality (AR), virtual reality (VR) and wearable tech. Join over 4,000 attendees from all over the world including a mix of CEOs, CTOs, designers, developers, creative agencies, futurists, analysts, investors, and top press in a fantastic opportunity to learn, inspire, partner, and experience first hand the most exciting industry of our times. See more at http://AugmentedWorldExpo.com
Building the Matrix: Your First VR App (SVCC 2016)Liv Erickson
The slides from my talk, Building The Matrix: Your First VR App at Silicon Valley Code Camp, Oct. 2016. Development, design, and sample projects for virtual reality applications.
There are many benefits of leveraging open source components to accelerate development of innovative applications and frameworks. In this session, projects will be showcased which have used OGRE and Qt to build specialized tools for multimedia creation in industries such as marketing and animation.
Presentation by Steve Streeting held during Qt Developer Days 2009.
http://qt.nokia.com/developer/learning/elearning
The talk shows developers how to get the most out of Unity when developing VR under the unique challenges of mobile platforms. The presenter shares the experience of achieving a VR version of the Ice Cave mobile demo where a number of advanced highly optimized rendering effects are running simultaneously.
Enabling Mobile Virtual Reality Gaming | Nizar RomdanJessica Tams
Delivered at Casual Connect Europe 2016
Virtual Reality has emerged as a major new hot topic in the past couple of years. This talk will provide an overview of the VR landscape with a focus on the mobile side listing the key players as well as the major use cases. The talk will finish by presenting ARM's activities to enable an exciting Mobile VR Gaming experience.
Greane Tree Technology CTO Joseph Payette gave our latest “Lunch & Learn” presentation. With the number of tools and frameworks for cross platform mobile application development increasing every year, it can be a challenge to determine the best fit technology for a mobile project. All of these tools and frameworks have their advantages and disadvantages, as they leverage different mechanisms for abstracting differences across mobile devices in an effort to provide a single platform for rapid application development. To bring order to the various options at hand, Joe reviewed mobile application architectures (native, hybrid, and HTML5), and explored and compared a few hybrid tools and frameworks, namely PhoneGap (www.phonegap.com), Appcelerator (www.appcelerator.com), and MoSync (www.mosync.com). Joe’s mobile application development presentation includes sample code for these three tools and frameworks.
The Lunch and Learn series is a regular event where we discuss topics of interest to our projects and clients. Last month, Chad Calhoun explored Git Interactive Techniques.
Philipp Nagele (Wikitude): Context Is for Kings: Putting Context in the Hands...AugmentedWorldExpo
A talk from the Develop Track at AWE USA 2018 - the World's #1 XR Conference & Expo in Santa Clara, California May 30- June 1, 2018.
Philipp Nagele (Wikitude): Context Is for Kings: Putting Context in the Hands of AR Developers
In this session, Philipp Nagele will explore why AR centers all around context and why contextual understanding is fundamental to any AR experience. He will show how Wikitude is trying to solve this problem for AR developers and provide technical details about the new release of the Wikitude SDK.
http://AugmentedWorldExpo.com
Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016Codemotion
This year for the Salone del Mobile at frog, we came up with a funky experiment, based on Drones, Android, and VR. In this talk, your hosts will walk you through our Drone Race experiment, touching topics like real-time computer vision, reactive programming for mobile, indoor positioning and (wheeled) Drones hacking. The variety and complexity of these topics is equal to its coolness though, so you may be puzzled asking yourself: "Where do I start?" We will share experiences and lots of code, so that you can start right away.
Get to know the Wikitude SDK in its details. Learn what the Wikitude SDK is offering and how it is structured. See the advantages of the Native API vs. the JavaScript API.
Augmented World Expo (AWE) is back for its seventh year in our largest conference and expo featuring technologies giving us superpowers: augmented reality (AR), virtual reality (VR) and wearable tech. Join over 4,000 attendees from all over the world including a mix of CEOs, CTOs, designers, developers, creative agencies, futurists, analysts, investors, and top press in a fantastic opportunity to learn, inspire, partner, and experience first hand the most exciting industry of our times. See more at http://AugmentedWorldExpo.com
Building the Matrix: Your First VR App (SVCC 2016)Liv Erickson
The slides from my talk, Building The Matrix: Your First VR App at Silicon Valley Code Camp, Oct. 2016. Development, design, and sample projects for virtual reality applications.
There are many benefits of leveraging open source components to accelerate development of innovative applications and frameworks. In this session, projects will be showcased which have used OGRE and Qt to build specialized tools for multimedia creation in industries such as marketing and animation.
Presentation by Steve Streeting held during Qt Developer Days 2009.
http://qt.nokia.com/developer/learning/elearning
The talk shows developers how to get the most out of Unity when developing VR under the unique challenges of mobile platforms. The presenter shares the experience of achieving a VR version of the Ice Cave mobile demo where a number of advanced highly optimized rendering effects are running simultaneously.
Enabling Mobile Virtual Reality Gaming | Nizar RomdanJessica Tams
Delivered at Casual Connect Europe 2016
Virtual Reality has emerged as a major new hot topic in the past couple of years. This talk will provide an overview of the VR landscape with a focus on the mobile side listing the key players as well as the major use cases. The talk will finish by presenting ARM's activities to enable an exciting Mobile VR Gaming experience.
Greane Tree Technology CTO Joseph Payette gave our latest “Lunch & Learn” presentation. With the number of tools and frameworks for cross platform mobile application development increasing every year, it can be a challenge to determine the best fit technology for a mobile project. All of these tools and frameworks have their advantages and disadvantages, as they leverage different mechanisms for abstracting differences across mobile devices in an effort to provide a single platform for rapid application development. To bring order to the various options at hand, Joe reviewed mobile application architectures (native, hybrid, and HTML5), and explored and compared a few hybrid tools and frameworks, namely PhoneGap (www.phonegap.com), Appcelerator (www.appcelerator.com), and MoSync (www.mosync.com). Joe’s mobile application development presentation includes sample code for these three tools and frameworks.
The Lunch and Learn series is a regular event where we discuss topics of interest to our projects and clients. Last month, Chad Calhoun explored Git Interactive Techniques.
Making Augmented Reality Applications with Android NDK
sdc-2016-gvrf-and-io_public
1. Open Source GearVR Framework (GVRf)
Intro, 3D Cursor, and Input Devices
Tom Flynn
Director, SW Engineering for VR – Mobile R&D Lab
Rick Lau
Director, Product Management & Engineering for VR Interaction/IOT – Mobile R&D Lab
2. 2
Agenda – Developing for the Gear VR
• Gear VR Framework
– Introduction
– How far we’ve come
– Scripting demonstration
– Planned Features
• Gear VR Framework Interaction
– 3D Cursor
– VR I/O Devices & Integrated Devices
3. 3
GVRf - Introduction
• Current development options for the Gear VR:
– Unity 3D plugin
– Unreal plugin
– Oculus Mobile SDK (C++ and OpenGL and Android NDK)
• What’s missing from the above options
– A Java based VR framework
– Something that is easy for non-game developers who still want to
develop VR apps
4. 4
GVRf - Introduction
• Java library that allows easy
development of VR applications in
Java for Android
• Built on top of Oculus Mobile SDK
and compliments existing solutions
• Allows developers to:
– Load common 3d file formats (fbx, collada,
obj, etc.)
– Perform scene management
– Attach animations to objects
– Automatically handle head tracking
– Automatically handle drawing the scene in
stereo
• Open Source; No royalties
6. 6
GVRf – Hello World App
// get a handle to the scene
GVRScene scene = gvrContext.getMainScene();
// load texture
GVRTexture texture = gvrContext.loadTexture(“GearVR.jpg”);
// create a scene object
GVRSceneObject sceneObject = new GVRSceneObject(gvrContext, 4.0f, 4.0f, texture);
// set the object position
sceneObject.getTransform().setPosition(0.0f, 0.0f, -3.0f);
// add the scene object to the scene graph
scene.addSceneObject(sceneObject);
7. 7
New GVRf Features – Standard Geometric Shapes
Quad Cube Text
Sphere Cylinder Cone
9. 9
New GVRf Features – Miscellaneous
Frustum culling
State sorting
Transparency sorting
Level of Detail
Full Assimp support
Skinning and skeleton animations
Alpha to Coverage
10. 10
New GVRf Features – Miscellaneous (cont)
Display and interact with Android Views
Android Studio support
Monoscopic support on non-Samsung phones
IO subsystem for VR Input
Scripting support for Lua and JavaScript
Support for multiple lights,
– lightmapping,
– and shadows
Contributions from KolorCompany, Niusounds, and many others.
Much much more
11. 11
GVRf Scripting Features – Easy to Do
• Support for either Lua or JavaScript
• Can write full program logic via scripting.
• Can attach scripts to scene objects (“script bundles”)
– See gvr-javascript sample.
• Connect over network and perform scripting commands
– gvrContext.startDebugServer();
• Remote scripting support:
– Compile and run the gvr-remote-scripting demo on
https://github.com/gearvrf/GearVRf-Demos/
– adb forward tcp:1645 tcp:1645
– telnet localhost 1645 (on Windows, use Window’s telnet program or cygwin).
12. 12
Scripting – Displaying a 360 Photosphere
// https://github.com/gearvrf/GearVRf-Demos/blob/scripting/
// gvr-remote-scripting/assets/photo.js
importPackage(org.gearvrf);
importPackage(org.gearvrf.scene_objects);
var url = new java.net.URL("https://raw.githubusercontent.com/gearvrf/GearVRf-
Demos/master/gvrf_minimal360photo/res/raw/photosphere.jpg");
var resource = new GVRAndroidResource(gvrf, url);
var texture = gvrf.loadFutureTexture(resource);
var photosphere = new GVRSphereSceneObject(gvrf, false, texture);
photosphere.getTransform().setScale(20, 20, 20);
photosphere.setName(“photosphere”);
var scene = gvrf.getMainScene();
scene.addSceneObject(photosphere);
13. 13
Scripting – Loading and Playing a 360 Video
// https://github.com/gearvrf/GearVRf-Demos/blob/scripting/
// gvr-remote-scripting/assets/video.js
importPackage(org.gearvrf);
importPackage(org.gearvrf.scene_objects);
var sphere = new GVRSphereSceneObject(gvrf, false);
var mesh = sphere.getRenderData().getMesh();
var mediaPlayer = new
org.gearvrf.utility.ImageUtils.createMediaPlayer("https://raw.githubusercontent.com/
gearvrf/GearVRf-Demos/master/gvrf_minimal360video/assets/videos_s_3.mp4")
var video = new GVRVideoSceneObject(gvrf, mesh, mediaPlayer,
GVRVideoSceneObject.GVRVideoType.MONO );
video.setName(“video”);
var scene = gvrf.getMainScene();
scene.addSceneObject(video);
mediaPlayer.start();
14. 14
Scripting – Loading Object Over Network and Animating It
// https://github.com/gearvrf/GearVRf-Demos/blob/scripting/
// gvr-remote-scripting/assets/trex.js
importPackage(org.gearvrf);
var scene = gvrf.getMainScene();
var trex_url = new java.net.URL("https://github.com/gearvrf/GearVRf-Demos/raw/
master/gvr-meshanimationsample/assets/TRex_NoGround.fbx");
var trex = gvrf.loadModelFromURL(trex_url);
scene.addSceneObject(trex);
trex.getTransform().setPosition(0.0, -5.0, -10.0);
trex.getTransform().setRotationByAxis(90.0, 1.0, 0.0, 0.0);
trex.getTransform().setRotationByAxis(40.0, 0.0, 1.0, 0.0);
trex.getTransform().setScale(1.5, 1.5, 1.5);
trex.setName(“trex”);
var engine = gvrf.getAnimationEngine();
var animation = trex.getAnimations().get(0);
animation.setRepeatMode(1).setRepeatCount(-1);
animation.start(engine);
15. 15
GVRf – Planned Features
• Support for OVR_Multiview
• Support for Vulkan
• Support for X3D
• Better support for physics and spatial audio
– Already have samples for jbullet and cardboard-audio
• Debugger and profiling information
• Lots of optimizations still to come
16. 16
GVRf – Available Now!
• Available as Open Source!
• Apache 2.0 license
• www.gearvrf.org
• https://github.com/Samsung/GearVRf
• https://github.com/gearvrf/GearVRf-Demos/
• Mailing lists: http://lists.gearvrf.org/
17. 3D Cursor – New Interaction Model for VR
& Partner Input Devices
Rick Lau
18. 18
Challenge: How to improve the VR Interaction experience.
• Team was tasked to improve the VR interaction experience.
• Today, gaze cursor is the primary input device.
• Expensive and complicated.
– Too much coding, too little results.
– 3rd party HW took a lot of time to integrate, per application.
– App developers had to understand and develop to specific HW.
19. 19
Solution - 3D Cursor!
A cursor for VR - extension of the PC
mouse/cursor.
Framework level implementation.
Standardizes interaction for VR apps
and input type integration.
Behavior
Same API as the PC cursor but with a
Z-axis component.
An unlimited number of cursors can be
instantiated.
Cursors can be dynamically instantiated
and destroyed.
Cursors can be customized.
3D Cursor
Input Device
GVRF
Scene Graph
&
Objects
Application
20. 20
1) Laser Cursor
– onCollision events for all objects in path of “laser”.
2) Point Cursor
– Mass and shape of object can be user defined.
– Intersection with other objects are more specific.
– Objects can be exlcuded from onCollision events.
2 Types of 3D Cursor
point cursor
object
intersect event/volume
optional: application
calculates intersected volume
21. 21
Point Cursor Example - Cannonball as a Cursor
cursor/cannonball’s parabolic (-like...)
trajectory can be calculated
onCollision detection of object,
explosion animation can be rendered
2
3
1 cursor is instantiated
22. 22
Point Cursor Example - Sword as a Point Cursor
onCollision Event
Cursor
onCollision animation
As volume of intersection
increases, shape of pumpkin can
change to reflect slicing action.
23. 23
Point Cursor Example - Hand as a Cursor Input Device
center of palm used for primary
(x,y,z) cursor placement
Each finger can be a point cursor:
• onCollision event can be
determined by all any fingertip
touching object.
• onButtonClick event can be
determined by a pinch gesture.
24. 24
Samsung GearVRf Partners for Input & General Support
3D Cursor
Input Device
GVRF
Scene Graph
&
Objects
Application
Gear 2/S2
Hand Recognition
Eye Tracking
6DOF Controllers
VR Gloves
Mouse, Gamepad, GearVR Touchpad
2 Cursor Types
Laser Cursor Examples
Point Cursor Examples
Integrated to GVRF!
25. 25
Samsung GVRf I/O Partners & Live Demo
See and experience these cools VR device input demos at the GVRf booth!
26. 26
Touchless hand & eye tracking example using point & laser cursors.
Laser Cursor
(The Eye Tribe)
Point Cursors
(Gestigon)
27. 27
3D Cursor API Overview
3D Cursor
Input Device
GVRF
Scene Graph
&
Objects
Application
Application Interfaces
Cursor Manager
Event Listener for Objects
Cursor Event: OnCollision
Button Events: up, down, pressed
Customization/Settings
read, modify, save
Gestures
I/O Device Interfaces
(x,y,z) position
Num buttons &button click events
Gestures (tbd)
OK, cancel
up, down, left, right
scroll up, scroll down,
others...
mouse
gamepad
gaze
built in support
30. 30
Release Plans for 3D Cursor
• Basic features to date – baseline APIs
– More complex cursor behaviors in the second release.
• Gather feedback from this Developer’s Conference
• Will push to gearvrf.org open source website – May/June.
Raffle giveaway by XImmerse at the GVRf booth!
Get chance to win their next gen 6DOF VR controllers.