• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
426 lecture 4: AR Developer Tools

426 lecture 4: AR Developer Tools



Lecture 4 from the COSC 426 graduate class on Augmented Reality. Taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury. August 1st 2012

Lecture 4 from the COSC 426 graduate class on Augmented Reality. Taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury. August 1st 2012



Total Views
Views on SlideShare
Embed Views



1 Embed 1

https://si0.twimg.com 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    426 lecture 4: AR Developer Tools 426 lecture 4: AR Developer Tools Presentation Transcript

    • COSC 426: Augmented Reality Mark Billinghurst mark.billinghurst@hitlabnz.org August 1st 2012 Lecture 4: AR Developer Tools
    • Low Level AR Libraries
    • Low Level AR Libraries  ARToolKit   Marker based tracking  FLARToolKit   Flash version of ARToolKit  SSTT   Simple Spatial Template Tracking  Opira   Robust Natural Feature Tracking
    • What is ARToolKit?  Marker Tracking Library for AR applications   Open Source, Multi-platform (Linux, Windows, MacOS)  Overlays 3D virtual objects on real markers   Uses single tracking marker   Determines camera pose information (6 DOF)  ARToolKit Websites http://www.hitl.washington.edu/artoolkit/ http://artoolkit.sourceforge.net/
    • How it Works
    • ARToolKit Software  ARToolKit version: 2.65 or later  Currently two license models   Open Source (GPL): ARToolKit 2.72   Commercial (ARToolWorks): ARToolKit 4.0  OS: Linux, Windows, MacOS X, iPhone/Android  Programming language: C  Related software   ARToolKit Professional: Commercial version   ARToolKitPlus: Advanced version   NyARToolkit: Java and C# version   FLARToolKit: Flash version
    • ARToolKit Family ARToolKit NFTARToolKit ARToolKit Plus JARToolKit (Java) ARToolKit (Symbian) FLARToolKit (Flash) NyToolKit - Java, C#, - Android, WM FLARManager (Flash)
    • ARToolKit Structure ARvideo.lib DirectShow  Three key libraries:   AR32.lib – ARToolKit image processing functions   ARgsub32.lib – ARToolKit graphics functions   ARvideo.lib – DirectShow video capture class
    • Additional Software  ARToolKit just provides tracking   For an AR application you’ll need more software  High level rendering library   Open VRML, Open Inventor, osgART, etc  Audio Library   Fmod, etc  Peripheral support
    • What does ARToolKit Calculate?   Position of makers in the camera coordinates  Pose of markers in the camera coordinates  Output format   3x4 matrix format to represent the transformation matrix from the marker coordinates to the camera coordinates
    • Coordinate Systems
    • From Marker To Camera  Rotation & Translation TCM : 4x4 transformation matrix from marker coord. to camera coord.
    • AR Application Development
    • An ARToolKit Application  Initialization   Load camera and pattern parameters  Main Loop   Step1. Image capture and display   Step2. Marker detection   Step3. Marker identification   Step4. Getting pose information   Step5. Object Interactions/Simulation   Step6. Display virtual objects  End Application   Camera shut down
    • Sample ARToolKit Applications Ex. 1: Simple video display Ex. 2: Detecting a marker Ex. 3: Using pattern Ex. 4: Getting a 3D information Ex. 5: Virtual object overlay
    • Ex 1: Simple Video Display  Program : sample1.c  Key points   Loop structure   Video image handling   Camera parameter handling   Window setup   Mouse and keyboard handling
    • Sample1.c – video initialization  Configure the video input vconf = <video configuration string>  Start video capture arVideoCapStart();  In init(), open the video arVideoOpen( vconf ); arVideoInqSize(&xsize, &ysize);  When finished, close the video path arVideoCapStop(); arVideoClose();
    • Changing Image Size  For input capture " vconf = “videoWidth=320,videoHeight=240"; Note – the camera must support this image size  For display argInit( &cparam, 1.5, 0, 0, 0, 0 );The second parameter means zoom ratio for display image size related to input image.
    • Graphics handling: libARgsub  Set up and clean up the graphics window void argInit( ARParam *cparam, double zoom, int fullFlag, int xwin, int ywin, int hmd_flag ); void argCleanup( void ); cparam: camera parameter zoom: zoom ratio fullFlag: 0: normal, 1: full screen mode Xwin, ywin: create small window for debug hmd_flag: 0: normal, 1: optical see-through mode
    • Sample1.c Main Functionmain()!{!!init();!!argMainLoop( mouseEvent, !! !keyEvent, mainLoop); !}!
    • Graphics handling: libARgsub  Go into the iterative cycle void argMainLoop( void (*mouseFunc)(int btn,int state,int x,int y), void (*keyFunc)(unsigned char key, int x, int y), void (*mainFunc)(void) );  Swap buffers void argSwapBuffers( void );
    • Sample1.c - mainLoop Function if( dataPtr = (ARUint8 *) arVideoGetImage()) == NULL ) { arUtilSleep(2); return; } argDrawMode2D(); argDispImage(dataPtr, 0, 0 ); arVideoCapNext(); argSwapBuffers();
    • Image capture: libARvideo  Return the pointer for captured image   ARUint8 *arVideoGetImage( void );  Pixel format and byte size are defined in config.h   #define AR_PIX_FORMAT_BGR   #define AR_PIX_SIZE 3
    • Graphics handling: libARgsub  Set the window for 2D drawing void argDrawMode2D( void );  Set the window for 3D drawing void argDrawMode3D( void ); void argDraw3dCamera( int xwin, int ywin );  Display image void argDispImage( ARUint8 *image, int xwin, int ywin );
    • Ex. 2: Detecting a Marker  Program : sample2.c  Key points   Threshold value   Important external variables   arDebug – keep thresholded image   arImage – pointer for thresholded image   arImageProcMode – use 50% image for image processing -  AR_IMAGE_PROC_IN_FULL -  AR_IMAGE_PROC_IN_HALF
    • Sample2.c – marker detection/* detect the markers in the video frame */if(arDetectMarker(dataPtr, thresh, &marker_info, &marker_num) < 0 ) { cleanup(); exit(0);}for( i = 0; i < marker_num; i++ ) { argDrawSquare(marker_info[i].vertex,0,0);}
    • Sample2.c – marker_info structure typedef struct { int area; int id; int dir; double cf; double pos[2]; double line[4][3]; double vertex[4][2]; } ARMarkerInfo; !
    • Ex. 3: Using a Pattern  Program : sample3.c  Key points   Pattern files loading   Structure of marker information -  Region features -  Pattern Id, direction -  Certainty factor   Marker identification
    • Making a pattern template  Use of utility program: mk_patt.exe  Show the pattern  Put the corner of red line segments on the left-top vertex of the marker  Pattern stored as a template in a file  1:2:1 ratio determines the pattern region used
    • Sample3.c – Pattern File Loadingint patt_id;char *patt_name = “Data/kanjiPatt”/* load pattern file */if(patt_id=arLoadPatt (patt_name) < 0){ printf ("Pattern file load error !! n"); exit(0);}
    • Checking for known patterns/* check for known patterns */ k = -1;for( i = 0; i < marker_num; i++ ) { if( marker_info[i].id == patt_id) { /* youve found a pattern */ printf("Found pattern: %d n",patt_id); if( k == -1 ) k = i; else /* make sure you have the best pattern (highest confidence factor) */ if( marker_info[k].cf < marker_info[i].cf ) k = i; }}
    • Ex. 4 – Getting 3D information  Program : sample4.c  Key points   Definition of a real marker   Transformation matrix -  Rotation component -  Translation component
    • Sample4.c – Transformation matrixdouble marker_center[2] = {0.0, 0.0};double marker_width = 80.0;double marker_trans[3][4];arGetTransMat(&marker_info[i], marker_center, marker_width, marker_trans);
    • Finding the Camera PositionThis function sets transformation matrix from marker to camera into marker_trans[3][4]." arGetTransMat(&marker_info[k], marker_center, marker_width, marker_trans);You can see the position information in the values of marker_trans[3][4]." " Xpos = marker_trans[0][3]; Ypos = marker_trans[1][3]; Zpos = marker_trans[2][3];
    • ARToolKit Coordinate Frame
    • Ex. 5- Virtual Object Display  Program : sample5.c  Key points   OpenGL parameter setting   Setup of projection matrix   Setup of modelview matrix
    • Appending your own OpenGL codeSet the camera parameters to OpenGL Projection matrix. argDrawMode3D(); argDraw3dCamera( 0, 0 );Set the transformation matrix from the marker to the camera to the OpenGL ModelView matrix. argConvGlpara(marker_trans, gl_para); glMatrixMode(GL_MODELVIEW); glLoadMatrixd( gl_para );After calling these functions, your OpenGL objects aredrawn in the real marker coordinates.
    • 3D CG Model Rendering  ARToolKit does not have a function to handle 3D CG models.  3rd party CG rendering software should be employed.   OpenVRML   OpenSceneGraph   etc
    • Loading Multiple Patterns  Sample File: LoadMulti.c   Uses object.c to load  Object Structure typedef struct { char name[256]; int id; int visible; double marker_coord[4][2]; double trans[3][4]; double marker_width; double marker_center[2]; } ObjectData_T;
    • Finding Multiple Transforms  Create object listObjectData_T *object;  Read in objects - in init( )read_ObjData( char *name, int *objectnum );  Find Transform – in mainLoop( )for( i = 0; i < objectnum; i++ ) { ..Check patterns ..Find transforms for each marker }
    • Drawing Multiple Objects  Send the object list to the draw functiondraw( object, objectnum );  Draw each object individuallyfor( i = 0; i < objectnum; i++ ) { if( object[i].visible == 0 ) continue; argConvGlpara(object[i].trans, gl_para); draw_object( object[i].id, gl_para);}
    • Limitations of ARToolKit  Partial occlusions cause tracking failure  Affected by lighting and shadows  Tracking range depends on marker size  Performance depends on number of markers   cf artTag, ARToolKitPlus  Pose accuracy depends on distance to marker  Pose accuracy depends on angle to marker
    • ARToolKit in the World  Hundreds of projects  Large research community
    • FLARToolKit  Flash AS3 Version of the ARToolKit (was ported from NyARToolkit the Java Version of the ARToolkit)  enables augmented reality in the Browser  uses Papervision3D for as 3D Engine  available at http://saqoosha.net/  dual license, GPL and commercial license
    • AR Application Components Adobe Flash Papervision 3D FLARToolkit
    • private function mainEnter(e:Event):void { /* Capture video frame*/ capture.draw(vid); /* Detect marker */ if (detector.detectMarkerLite(raster, 80) && detector.getConfidence() > 0.5) { //Get the transfomration matrix for the current marker position detector.getTransformMatrix(trans); //Translates and rotates the mainContainer so it looks right mainContainer.setTransformMatrix(trans); //Render the papervision scene renderer.render(); } }
    • FLARToolKit Examples
    • Papervision 3D  http://www.papervision3d.org/  Flash-based 3D-Engine  Supports   import of 3D Models   texturing   animation   scene graph  alternatives: Away3d, Sandy,…
    • Source Packages  „Original“ FLARToolkit (Libspark, Saqoosha) ( http://www.libspark.org/svn/as3/FLARToolKit/trunk/ )  Start-up-guides   Saqoosha (http://saqoosha.net/en/flartoolkit/start-up-guide/ )   Miko Haapoja (http://www.mikkoh.com/blog/?p=182 )  „Frameworks“   Squidder MultipleMarker – Example ( http://www.squidder.com/2009/03/06/flar-how-to-multiple-instances-of-multiple- markers/ )   FLARManager (http://words.transmote.com/wp/flarmanager/ )
    • Other Languages  NyARToolKit   http://nyatla.jp/nyartoolkit/wp/   AS3, Java, C#, Processing, Unity, etc  openFrameworks   http://www.openframeworks.cc/   https://sites.google.com/site/ofauckland/examples/8-artoolkit-example   Support for other libraries -  Kinect, Audio, Physics, etc
    • void testApp::update(){ //capture video and detect markers mov.update(); if (mov.isFrameNew()) { img.setFromPixels(mov.getPixels(), ofGetWidth(), ofGetHeight()); gray = img; tracker.setFromPixels(gray.getPixels()); }}//--------------------------------------------------------------void testApp::draw(){ //draw AR objects ofSetColor(0xffffff); mov.draw(0, 0); for (int i=0; i<tracker.markers.size(); i++) { ARMarkerInfo &m = tracker.markers[i]; tracker.loadMarkerModelViewMatrix(m); ofSetColor(255, 255, 0, 100); ofCircle(0,0,25); ofSetColor(0); ofDrawBitmapString(ofToString(m.id),0,0); }}
    • AR Tools
    • Building Compelling AR Experiences experiences applications tools Authoring components Tracking, Display Sony CSL © 2004
    • AR Authoring  Software Libraries   osgART, Studierstube, MXRToolKit  Plug-ins to existing software   DART (Macromedia Director), mARx, Unity,  Stand Alone   AMIRE, BuildAR, etc  Next Generation   iaTAR (Tangible AR)
    • mARx Plug-in  3D Studio Max Plug-in  Can model and view AR content at the same time
    • BuildAR  http://www.buildar.co.nz/  Stand alone application  Visual interface for AR model viewing application  Enables non-programmers to build AR scenes
    • Plug-ins for 3D authoring tools  Unity - Game development tool (www.unity3d.com)  Esperient Creator – General interactive 3D authoring (www.esperient.com)
    • Metaio Design  Complete commercial authoring platform   http://www.metaio.com/software/design/   Offers viewer and editor tools   Drag and drop tools   3D model import
    • Total Immersion D’Fusion Studio  Complete commercial authoring platform   http://www.t-immersion.com/   Multi-platform   Markerless tracking   Scripting   Face tracking   Finger tracking   Kinect support
    • Others  AR-Media   http://www.inglobetechnologies.com/   Google sketch-up plug-in  LinceoVR   http://linceovr.seac02.it/   AR/VR authoring package  Libraries   JARToolKit, MXRToolKit, ARLib, Goblin XNA
    • OSGART Programming Library  Integration of ARToolKit with a High-Level Rendering Engine (OpenSceneGraph) OSGART= OpenSceneGraph + ARToolKit  Supporting Geometric + Photometric Registration
    • osgART:Features  C++ (but also Python, Lua, etc).  Multiple Video Input supports:   Direct (Firewire/USB Camera), Files, Network by ARvideo, PtGrey, CVCam, VideoWrapper, etc.  Benefits of Open Scene Graph   Rendering Engine, Plug-ins, etc
    • Advanced Authoring: iaTAR (Lee 2004)  Immersive AR Authoring  Using real objects to create AR applications
    • More Information•  Mark Billinghurst –  mark.billinghurst@hitlabnz.org •  Websites –  www.hitlabnz.org