Creating Novel Augmented Reality  Applications for Mobile Users                    www.diotasoft.comSebastian Knödel      ...
ContentBrief History of Mobile ARMobile AR SDKs & APIsiCyclope for IPhoneQualcomm AR SDK (QCAR)QCAR & Unity3DMobile AR Gam...
What is mobile AR good for?        Services                 - Information visualization                 - Navigation      ...
Mobile Augmented RealityMilgrams Continuum                        Mixed Reality     Real      Augmented                  A...
Whats the problem?Where am I ?               Breizhcamp   5
Brief History of Mobile ARSensor based tracking   GPS position   Compass orientation                                      ...
Vision-based TrackingCalculate camera position and orientation relative tomarkers – image registration1. Stage            ...
ARToolkit  Breizhcamp   8
Natural feature-based tracking             Breizhcamp     9
Brief History of Mobile AR            Vision based tracking                                     Courtesy Qualcomm         ...
iCylcopeVideo & Sound    Capture Camera & Tracking                       Rendering OpenGL ES                Breizhcamp    ...
Video & Sound             Sound mp3                 Video pvrvideo = [Video alloc];for (int i = 0; i < countFile; i++){   ...
Camera Capture & Tracking Capture camera image/*We setup the input*/AVCaptureDeviceInput *captureInput = [AVCaptureDeviceI...
Rendering OpenGL ES    initTexture();    updateTexture();    updatePlaneForVideo();updateTexture(){      NSData * data = [...
Mobile AR SDKs & APIsProprietary                Public              Breizhcamp            15
Qualcomm AR SDK (QCAR)Different target types                         Breizhcamp   16
Workflow QCARTarget Images                             Mobile application                               App Logic &       ...
QCAR APIVirtual   Multi Image      ImageButton      Target         Target                                       Marker    ...
Developing with QCARmyAR_App.java/** The main activity for the myAR_App */public class myAR_App extends Activity{…}      m...
Developing with QCARmyAR_App/** The main activity for the myAR_App */public class myAR_App extends Activity{     protected...
Developing with QCARmyAR_GLView/** Configuring an translucent OpenGL surface view */public class myAR_GLView extends GLSur...
Developing with QCARmyAR_Renderer/** The renderer class for the myAR_App activity. */public class myAR_Renderer implements...
Developing with QCARmyAR_NativeRenderer.cppmyRenderFrameNative(JNIEnv *, jobject){     glClear(…);     // Render video bac...
Developing with QCAR        Breizhcamp     24
QCAR and Unity3D      Breizhcamp   25
QCAR and Unity3D                   Workflow                   •    Create Target QDevNet                   •    Add target...
Sound-plugin for Unity3D                                JavaActivity.java SDK                             SoundManager.jav...
Sound-plugin for Unity3D        public class JavaClass{	     JavaActivity.java                                 SoundManage...
Diotasoft GamesThe Circus                ARLabyrinth             Breizhcamp                 29
Conclusions     QCAR Android                             QCAR Unity       SDK/NDK                                       • ...
Future is Markerless        Breizhcamp     31
Merci beaucoup     Breizhcamp   32
Upcoming SlideShare
Loading in …5
×

Breizhcamp Rennes 2011

1,642 views
1,538 views

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,642
On SlideShare
0
From Embeds
0
Number of Embeds
305
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Breizhcamp Rennes 2011

  1. 1. Creating Novel Augmented Reality Applications for Mobile Users www.diotasoft.comSebastian Knödel Ludovic Perrierskn@diotasoft.com lpe@diotasoft.comsekont.tumblr.com ludovic_perrier sekond Breizhcamp 1
  2. 2. ContentBrief History of Mobile ARMobile AR SDKs & APIsiCyclope for IPhoneQualcomm AR SDK (QCAR)QCAR & Unity3DMobile AR GamesFuture of mobile AR Breizhcamp 2
  3. 3. What is mobile AR good for? Services - Information visualization - Navigation Games Promotion, Advertising Industrial Applications Courtesy Qualcomm Why now? Because todays Smart Phones are able to! - Hardware (computing capacity, sensors, cameras, screensize, …) - Software, powerful SDKs (Android, Iphone, QCAR, …)Courtesy Layar Breizhcamp 3
  4. 4. Mobile Augmented RealityMilgrams Continuum Mixed Reality Real Augmented Augmented Virtual Environment Environment Virtuality Environment Breizhcamp 4
  5. 5. Whats the problem?Where am I ? Breizhcamp 5
  6. 6. Brief History of Mobile ARSensor based tracking GPS position Compass orientation Courtesy Wikitude Wikitude 2009Vision based tracking Courtesy TU Graz Special Marker 1999 ARToolkitPlus 2007 NyARToolkit 2009 Breizhcamp 6
  7. 7. Vision-based TrackingCalculate camera position and orientation relative tomarkers – image registration1. Stage 2. Stage Feature Detection Coordinate system Breizhcamp 7
  8. 8. ARToolkit Breizhcamp 8
  9. 9. Natural feature-based tracking Breizhcamp 9
  10. 10. Brief History of Mobile AR Vision based tracking Courtesy Qualcomm Qualcomm AR Courtesy Metaio Metaio/JuneioDiotasoft – iCyclope – Werber 2010 Breizhcamp 10
  11. 11. iCylcopeVideo & Sound Capture Camera & Tracking Rendering OpenGL ES Breizhcamp 11
  12. 12. Video & Sound Sound mp3 Video pvrvideo = [Video alloc];for (int i = 0; i < countFile; i++){ [imgDataArray addObject:[[NSData alloc] initWithContentsOfFile:path]];} Breizhcamp 12
  13. 13. Camera Capture & Tracking Capture camera image/*We setup the input*/AVCaptureDeviceInput *captureInput = [AVCaptureDeviceInput …];/*We setup the output*/AVCaptureVideoDataOutput *captureOutput = [AVCaptureVideoDataOutput alloc] ..]; From image to positionbool mTracked = [tracking imgPointer width:width height:height :position]; Breizhcamp 13
  14. 14. Rendering OpenGL ES initTexture(); updateTexture(); updatePlaneForVideo();updateTexture(){ NSData * data = [mVideo getCurrentImage]; glCompressedTexImage2D(GL_TEXTURE_2D, …, [data bytes])} Breizhcamp 14
  15. 15. Mobile AR SDKs & APIsProprietary Public Breizhcamp 15
  16. 16. Qualcomm AR SDK (QCAR)Different target types Breizhcamp 16
  17. 17. Workflow QCARTarget Images Mobile application App Logic & Rendering Engine Target Position QDevNet QCAR library Target Resources Breizhcamp 17
  18. 18. QCAR APIVirtual Multi Image ImageButton Target Target Marker Developer Application Tracker Camera Rendering Android Breizhcamp 18
  19. 19. Developing with QCARmyAR_App.java/** The main activity for the myAR_App */public class myAR_App extends Activity{…} myAR_Renderer.java /** The renderer class for the myAR_App activity. */ public class myAR_Renderer implements GLSurfaceView.Renderer{…} myAR_GLView.java /** Configuring an translucent OpenGL surface view */ public class myAR_GLView extends GLSurfaceView{…} myAR_NativeRenderer.cpp /** OpenGL Rendering */ myNativeRenderFrame(JNIEnv *, jobject){…} Breizhcamp 19
  20. 20. Developing with QCARmyAR_App/** The main activity for the myAR_App */public class myAR_App extends Activity{ protected void onCreate(Bundle savedInstanceState) { loadTextures(); // init myAR_GLView, init myAR_Renderer, ScreenOrientation, SplashScreen initApplication(); mInitQCARTask = new InitQCARTask(); // Initialize QCAR asynchronously. mInitQCARTask.execute(); mLoadTrackerTask = new LoadTrackerTask(); // Load the tracking data mInitQCARTask.execute(); onQCARInitializedNative();//tracking of multiple targets simultaneously startCamera(); }} Breizhcamp 20
  21. 21. Developing with QCARmyAR_GLView/** Configuring an translucent OpenGL surface view */public class myAR_GLView extends GLSurfaceView{ public void init(int flags, boolean translucent, int depth, int stencil) { this.getHolder().setFormat(PixelFormat.TRANSLUCENT); setOpenGLContext(1.1/2.0); }} Breizhcamp 21
  22. 22. Developing with QCARmyAR_Renderer/** The renderer class for the myAR_App activity. */public class myAR_Renderer implements GLSurfaceView.Renderer{ public void onSurfaceCreated(GL10 gl, EGLConfig config) { initRendering(); // native fct QCAR.onSurfaceCreated(); } /** Called when the surface changed size. */ public void onSurfaceChanged(GL10 gl, int width, int height) { updateRendering(width, height); // native fct QCAR.onSurfaceChanged(width, height); } /** Called to draw the current frame. */ public void onDrawFrame(GL10 gl) { // Call native function to render content myRenderFrameNative(); }} Breizhcamp 22
  23. 23. Developing with QCARmyAR_NativeRenderer.cppmyRenderFrameNative(JNIEnv *, jobject){ glClear(…); // Render video background: QCAR::Renderer::getInstance().begin(); for(i … state.getNumActiveTrackables()) { QCAR::Trackable* trackable = state.getActiveTrackable(i); modelViewMatrix = convertPose2GLMatrix(trackable->getPose()); glMatrixMode(GL_MODELVIEW); glLoadMatrixf(modelViewMatrix.data); // Add Your OpenGLES 1.1/2.0 Render Code Here drawTeapot(); } // finish GL Rendering QCAR::Renderer::getInstance().end();} Breizhcamp 23
  24. 24. Developing with QCAR Breizhcamp 24
  25. 25. QCAR and Unity3D Breizhcamp 25
  26. 26. QCAR and Unity3D Workflow •  Create Target QDevNet •  Add target resources to project •  Add ARCamera Prefab •  Add myTarget Prefab •  Add 3D objects, physics, shaders, Game Logic, etc. •  Deploy on Android device Breizhcamp 26
  27. 27. Sound-plugin for Unity3D JavaActivity.java SDK SoundManager.java NDK NativeJavaBridge.CPP (JNI)Unity3D AndroidSoundPlugin.CS (C#) Breizhcamp 27
  28. 28. Sound-plugin for Unity3D public class JavaClass{ JavaActivity.java SoundManager.java private Activity mActivity; public static void initSounds(Context theContext) public JavaClass(Activity currentActivity) { mActivity = currentActivity; { SoundManager.initSounds(mActivity); … SDK } mAudioManager = (AudioManager) mContext.getSystemService public int ActivitySoundPlay(int id, (Context.AUDIO_SERVICE); float volume) { } return SoundManager.playSound(id, volume,…); } jint JNI_OnLoad(JavaVM* vm, void* reserved) { NativeJavaBridge.CPP (JNI) jclass cls_Activity = jni_env->FindClass("com/unity3d/player/UnityPlayer"); jclass cls_JavaClass = jni_env->FindClass("com/diotasoft/pluginunity/JavaClass"); activitySoundPlay = jni_env->GetMethodID(cls_JavaClass, "ActivitySoundPlay", "(IFFIF)I"); … } NDK int playSound(int index, float volume, int repeat, float pitch) { jint currentStream = (jint)jni_env->CallObjectMethod(JavaClass, activitySoundPlay, index, volume, repeat, pitch); … } public class AndroidSoundPoolPlugin : MonoBehaviour { AndroidSoundPlugin.CS (C#) [DllImport("javabridge")] Unity private static extern int playSound(int index, float volume, int repeat, float pitch); [DllImport("javabridge")] private static extern void stopSound(int currentStream); }; Breizhcamp 28
  29. 29. Diotasoft GamesThe Circus ARLabyrinth Breizhcamp 29
  30. 30. Conclusions QCAR Android QCAR Unity SDK/NDK • Closed system• Open system (hands on) • Fast content creation• Existant code integration • Plugins & profiling• Features you implement • Rich feature sets provided• Developer community Breizhcamp 30
  31. 31. Future is Markerless Breizhcamp 31
  32. 32. Merci beaucoup Breizhcamp 32

×