Philipp Nagele (CTO, Wikitude GmbH) gives an Insider Deep-Dive into the Wikitude SDK An introduction into the many options of the Wikitude SDK with a deeper look into advanced features like Plugins API and how to combine third party libraries with the Wikitude SDK. We will look into the general architecture of the SDK and deep-dive into a few outstanding (and maybe not so well-known) features of the SDK.

1. Deep-Dive into the Wikitude SDK

2. Who is talking
2
Philipp Nagele
● CTO of Wikitude (Product & Technology)
● MSc in Computer System Engineering
from Halmstad University in Sweden
● Previous companies:
－ 3united/VeriSign Inc.
－ T-Mobile Austria

3. Agenda for today
● Who is Wikitude
● Architecture of the Wikitude SDK
● A deeper look at the JavaScript API and the Plugins API

4. v v v
Wikitude AR at a glance.
First AR app
in the world
in 2008
10,000 apps
powered by
Wikitude SDK
3D tracking
technology
v
REFERENCES

5. 5
HQ office in Salzburg, Austria

6. 6

7. AR creator and CMSCloud Recognition
Wikitude AR Products & Tools
Connect your own CMS

8. Mock-ups only
… understand what you are looking at
Mock-ups only

9. … make the World your playground

10. ….augment books
Mock-ups only

11. … augment print, ads, catalogues, packaging

12. … virtually mount products

13. Maintenance

15. Platform agnostic technology
15
AR/VR wearables TabletsSmartphones
multi platform technology core based on C++

16. Tailored SDKs for Smart Glasses
Partnership since 2014,
optimized SDK for BT-200, full
documentation
→BT-300 in pipeline
Partnership since 2016
optimized SDK for R-7
Partnership since 2014, SDK for
M100, full documentation
→M300 in pipeline

17. Native API JavaScript API
Easy access to AR mobile development
Xamarin
PhoneGap
Cordova
titaniumunity
Android
iOS

18. Wikitude SDK Architecture
2D On-Device
Reco &
Tracking
2D Cloud
Reco
3D Reco &
Tracking
Unity3D
Unity
Plugin
Plugins API
Custom CV
plugin
Custom
camera
plugins
Your
App
Your
App Cordova
Plugin
Titanium
Module
Xamarin
Component
Your
App
Cordova Titanium Xamarin
Your
App
Your
App
JavaScript APINative API
Your
App
3D
Rendering
Augmen-
tations
LBS
Plugin
Manager
OpenGL ES2
Rendering
ARMv7
NEON
optimizations
ARMv8
optimizations
Camera
GPU
optimizations
OpenCL
Metal API
IMU
Hardware Abstraction Hardware Optimizations
C++ Layer
Java/Obj C

19. Wikitude JS SDK Java API
Wikitude JS SDK has a very slim Java API
• ArchitectView
• Lifecycle events (onCreate, onPause, onResume, … methods)
• Load Architect Worlds
Everything else is optional
• ArchitectUrlListener
• ArchitectWorldLoadedListener
• CaptureScreenCallback

20. Architect view composition
20
OpenGL ES 2 view
* Renders camera stream
* Renders augmentations
Web view
* Loads main .html file
* Transparent background

21. JavaScript Augmented Reality
Experience
21
Architect Worlds consist of:
• .html file(s)
• .js file(s)
• .css file(s)
• Defines your augmented reality experience
• Use the Wikitude JS API provided within the Architect view context
<script src=“https://wikitude.com/libs/architect.js"></script>
• Loaded by the Wikitude SDKs Architect view

22. Architect world
22
Javascript execution triggers actions in the underlying C++ layer

23. Types of augmentations
23
Augmentations are represented by AR.Drawable subclasses
• Images - AR.ImageResource / AR.ImageDrawable
• Videos (also with alpha channel) - AR.VideoDrawable
• 3D models - AR.Model
• Web views - AR.HTMLDrawable
• Labels - AR.Label

24. Wikitude JavaScript API
24
Model
• AR.GeoLocation
• AR.RelativeLocation
• AR.Tracker
• AR.ImageResource
View
• AR.ImageDrawable
• AR.VideoDrawable
• AR.Model
• AR.Label
Controller
• AR.GeoObject
• AR.Trackable2D
• AR.PropertyAnimation
• AR.AnimationGroup

25. Geo AR
25
How to define e.g. a point
of interest (POI)
• Define where the
augmentation is placed by
creating AR.GeoLocations
(latitude/longitude coordinates)
• Define the augmentation by
creating AR.Drawables
• Combine location and
drawables using
AR.GeoObjects
= The same geo object is displayed at different locations
POI C
POI A
POI A
Poi B

26. Pokémon Go like Geo AR
26
How to build a Pokémon Go like AR experience:

27. Pokémon Go like Geo AR
27
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
//a relative location being 12 meters south, 20 meters west and 13 meters lower than current position of the user
var relativeLocation = new AR.RelativeLocation(null, -12, -20, -13);

28. Pokémon Go like Geo AR
28
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
• Absolute to the user using AR.GeoLocation
//latitude, longitude, altitude
var location = new AR.GeoLocation(47.77317, 13.069929, 320.);

29. Pokémon Go like Geo AR
29
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon
//create a new Model
var model = new AR.Model(“models/pokemon.wt3”);

30. Pokémon Go like Geo AR
30
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon
• Can be animated through .fbx defined animations
//We want to start an animation inside a model
var model = new AR.Model(“models/pokemon.wt3");
var modelAnim = new AR.ModelAnimation(model, "hit"); //start hit-animation defined inside model

31. Pokémon Go like Geo AR
31
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• …
• AR.Model object to render the 3D model of a Pokémon
• …
• AR.GeoObject to actually bring the model onto the screen
// a GeoObject with drawables set on creation time
var geoObject = new AR.GeoObject(location, {
drawables : {
cam: [model] //the drawable representing the GeoObject in the camera view
}
});

32. Pokémon Go like Geo AR
32
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
geoObject.onEnterFieldOfVision = function() {
/* your custom behaviour e.g. start a certain model animation */
};
geoObject.onExitFieldOfVision = function() {
/* your custom behaviour */
};

33. Pokémon Go like Geo AR
33
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
var imageResource = new AR.ImageResource(“assets/pokemonIndicator.png”);
var pokemonIndicator = new AR.ImageDrawable(imageResource, 0.2);
// a GeoObject with drawables set on creation time
var geoObject = new AR.GeoObject(location, {
drawables : {
cam: [model], //the drawable representing the GeoObject in the camera view
indicator: pokemonIndicator
}
});

34. Pokémon Go like Geo AR
34

35. GeoObjects & HTML content
Menu bar
jQuery support
Geo-located POIs
Multiple Drawables
- 3 images
- 3 text labels
Web View
Augmented Reality View

36. Pokémon Go like Geo AR
36
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
• Use the AR.Radar component to permanently show the position of
a Pokémon in the AR scene

37. Pokémon Go like Geo AR
37

38. Pokémon Go like Geo AR
38
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
• Use the AR.Radar to permanently show the position of a Pokémon in
the AR scene
• Capture a screenshot after you caught a Pokémon using the
corresponding ArchitectView Java API.

39. Image Recognition
Augment target images using
JavaScript API capabilities (Drawables,
Sounds, Animations,Videos, 3D
Models)
var kitchenTracker = new AR.Tracker("Kitchen.wtc");
var overlay = new AR.VideoDrawable("howdens.mp4", 0.65);
var trackable2DObject = new AR.Trackable2DObject(
kitchenTracker: "Kitchen",
{ drawables:
{ cam: overlay
}});

40. Wikitude SDK Architecture
2D On-Device
Reco &
Tracking
2D Cloud
Reco
3D Reco &
Tracking
Unity3D
Unity
Plugin
Plugins API
Custom CV
plugin
Custom
camera
plugins
Your
App
Your
App Cordova
Plugin
Titanium
Module
Xamarin
Component
Your
App
Cordova Titanium Xamarin
Your
App
Your
App
JavaScript APINative API
Your
App
3D
Rendering
Augmen-
tations
LBS
Plugin
Manager
OpenGL ES2
Rendering
ARMv7
NEON
optimizations
ARMv8
optimizations
Camera
GPU
optimizations
OpenCL
Metal API
IMU
Hardware Abstraction Hardware Optimizations
C++ Layer
Java/Obj C

41. Plugins API in detail
41
Custom
Frame
Provider
Image
Frame
Standard
Frame
Provider
Wikitude
CV EngineRendering
Custom CV
Plugin
Plugins API
class Plugin { public:
…
void cameraFrameAvailable(const Frame&;
cameraFrame_);
void update(const vector<RecognizedTarget>
recognizedTargets_);
void startRender();
void endRender
…
};
1
2
3
4
5
6
7
class RecognizedTarget {
public:
const string&amp; getIdentifier() const;
const Mat4&amp; getModelViewMatrix()
const;
const Mat4&amp; getProjectionMatrix() const;
const float getDistanceToCamera() const;
};

42. Use-Cases for Plugins API
● Run own computer-vision related code in parallel to Wikitude SDK
－OCR Engine
－Barcode Reader
－Face Recognition
● Own the camera stream and the camera rendering
－Custom effects and shaders
－Remote camera stream with local processing

43. ….architecture, real estate….. ….gaming, filming industry….. ….indoor navigation, retail…..
New algorithms and concepts
3D Tracking / SLAM
Object Reco & Tracking

44. See more.
At Booth B6