COSC 426: Augmented Reality           Mark Billinghurst     mark.billinghurst@hitlabnz.org            August 1st 2012     ...
Low Level AR Libraries
Low Level AR Libraries  ARToolKit    Marker based tracking  FLARToolKit    Flash version of ARToolKit  SSTT    Simpl...
What is ARToolKit?  Marker Tracking Library for AR applications     Open Source, Multi-platform (Linux, Windows, MacOS)...
How it Works
ARToolKit Software  ARToolKit version: 2.65 or later  Currently two license models     Open Source (GPL): ARToolKit 2.7...
ARToolKit Family                                   ARToolKit NFTARToolKit                           ARToolKit Plus        ...
ARToolKit Structure                                      ARvideo.lib                                      DirectShow  Thr...
Additional Software  ARToolKit just provides tracking    For an AR application you’ll need more software  High level re...
What does ARToolKit Calculate?                              	  Position of makers in the camera coordinates  Pose of mar...
Coordinate Systems
From Marker To Camera  Rotation & Translation    TCM : 4x4 transformation matrix         from marker coord. to camera coo...
AR Application Development
An ARToolKit Application   Initialization     Load camera and pattern parameters  Main Loop     Step1. Image capture a...
Sample ARToolKit Applications Ex. 1: Simple video display Ex. 2: Detecting a marker Ex. 3: Using pattern Ex. 4: Gettin...
Ex 1: Simple Video Display  Program : sample1.c  Key points    Loop structure    Video image handling    Camera param...
Sample1.c – video initialization  Configure the video input   vconf = <video configuration string>  Start video capture ...
Changing Image Size  For input capture "   vconf = “videoWidth=320,videoHeight=240";   Note – the camera must support thi...
Graphics handling: libARgsub  Set up and clean up the graphics window   void argInit( ARParam *cparam, double zoom,      ...
Sample1.c Main Functionmain()!{!!init();!!argMainLoop( mouseEvent,     !!   !keyEvent, mainLoop);           !}!
Graphics handling: libARgsub  Go into the iterative cycle  void argMainLoop(     void (*mouseFunc)(int btn,int state,int ...
Sample1.c - mainLoop Function if( dataPtr = (ARUint8 *) arVideoGetImage()) == NULL ) {     arUtilSleep(2);     return; } a...
Image capture: libARvideo  Return the pointer for captured image    ARUint8 *arVideoGetImage( void );  Pixel format and...
Graphics handling: libARgsub  Set the window for 2D drawing   void argDrawMode2D( void );  Set the window for 3D drawing...
Ex. 2: Detecting a Marker  Program : sample2.c  Key points    Threshold value    Important external variables    arDe...
Sample2.c – marker detection/* detect the markers in the video frame */if(arDetectMarker(dataPtr, thresh,  &marker_info, &...
Sample2.c – marker_info structure  typedef struct {    int area;    int id;    int dir;    double cf;    double pos[2];   ...
Ex. 3: Using a Pattern  Program : sample3.c  Key points    Pattern files loading    Structure of marker     informatio...
Making a pattern template  Use of utility program:        mk_patt.exe  Show the pattern  Put the corner of red line   s...
Sample3.c – Pattern File Loadingint patt_id;char *patt_name = “Data/kanjiPatt”/* load pattern file */if(patt_id=arLoadPatt...
Checking for known patterns/* check for known patterns */    k = -1;for( i = 0; i < marker_num; i++ ) {    if( marker_info...
Ex. 4 – Getting 3D information  Program : sample4.c  Key points    Definition of a real marker    Transformation matri...
Sample4.c – Transformation matrixdouble marker_center[2] = {0.0, 0.0};double marker_width = 80.0;double marker_trans[3][4]...
Finding the Camera PositionThis function sets transformation matrix from marker  to camera into marker_trans[3][4]."  arGe...
ARToolKit Coordinate Frame
Ex. 5- Virtual Object Display  Program : sample5.c  Key points     OpenGL parameter setting     Setup of projection ma...
Appending your own OpenGL codeSet the camera parameters to OpenGL Projection matrix.   argDrawMode3D();   argDraw3dCamera(...
3D CG Model Rendering  ARToolKit does not have a function to handle   3D CG models.  3rd party CG rendering software sho...
Loading Multiple Patterns  Sample File: LoadMulti.c     Uses object.c to load  Object Structure   typedef struct {     ...
Finding Multiple Transforms  Create object listObjectData_T        *object;  Read in objects - in init( )read_ObjData( c...
Drawing Multiple Objects  Send the object list to the draw functiondraw( object, objectnum );  Draw each object individu...
Limitations of ARToolKit  Partial occlusions cause tracking failure  Affected by lighting and shadows  Tracking range d...
ARToolKit in the World  Hundreds of projects  Large research community
FLARToolKit  Flash AS3 Version of the ARToolKit     (was ported from NyARToolkit the Java Version of the ARToolkit)    e...
AR Application Components                   Adobe Flash                   Papervision 3D                   FLARToolkit
private function mainEnter(e:Event):void {              /* Capture video frame*/              capture.draw(vid);          ...
FLARToolKit Examples
Papervision 3D  http://www.papervision3d.org/  Flash-based 3D-Engine  Supports    import of 3D Models    texturing  ...
Source Packages  „Original“ FLARToolkit (Libspark, Saqoosha) (  http://www.libspark.org/svn/as3/FLARToolKit/trunk/ )  St...
Other Languages  NyARToolKit    http://nyatla.jp/nyartoolkit/wp/    AS3, Java, C#, Processing, Unity, etc  openFramewo...
void testApp::update(){ //capture video and detect markers    mov.update();    if (mov.isFrameNew()) {        img.setFromP...
AR Tools
Building Compelling AR Experiences          experiences          applications             tools       Authoring          c...
AR Authoring  Software Libraries      osgART, Studierstube, MXRToolKit  Plug-ins to existing software      DART (Macro...
mARx Plug-in  3D Studio Max Plug-in  Can model and view AR content at the same time
BuildAR    http://www.buildar.co.nz/    Stand alone application    Visual interface for AR model viewing application  ...
Plug-ins for 3D authoring tools  Unity - Game development tool (www.unity3d.com)  Esperient Creator – General interactiv...
Metaio Design  Complete commercial authoring platform    http://www.metaio.com/software/design/    Offers viewer and ed...
Total Immersion D’Fusion Studio  Complete commercial authoring platform    http://www.t-immersion.com/    Multi-platfor...
Others  AR-Media     http://www.inglobetechnologies.com/     Google sketch-up plug-in  LinceoVR     http://linceovr.s...
OSGART Programming Library  Integration of ARToolKit with a High-Level   Rendering Engine (OpenSceneGraph)  OSGART= OpenS...
osgART:Features  C++ (but also Python, Lua, etc).  Multiple Video Input supports:     Direct (Firewire/USB Camera), Fil...
Advanced Authoring: iaTAR (Lee 2004)  Immersive AR Authoring  Using real objects to create AR applications
More Information•  Mark Billinghurst	   –  mark.billinghurst@hitlabnz.org	•  Websites	   –  www.hitlabnz.org
Upcoming SlideShare
Loading in …5
×

426 lecture 4: AR Developer Tools

2,384 views

Published on

Lecture 4 from the COSC 426 graduate class on Augmented Reality. Taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury. August 1st 2012

Published in: Technology, Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,384
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
130
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

426 lecture 4: AR Developer Tools

  1. 1. COSC 426: Augmented Reality Mark Billinghurst mark.billinghurst@hitlabnz.org August 1st 2012 Lecture 4: AR Developer Tools
  2. 2. Low Level AR Libraries
  3. 3. Low Level AR Libraries  ARToolKit   Marker based tracking  FLARToolKit   Flash version of ARToolKit  SSTT   Simple Spatial Template Tracking  Opira   Robust Natural Feature Tracking
  4. 4. What is ARToolKit?  Marker Tracking Library for AR applications   Open Source, Multi-platform (Linux, Windows, MacOS)  Overlays 3D virtual objects on real markers   Uses single tracking marker   Determines camera pose information (6 DOF)  ARToolKit Websites http://www.hitl.washington.edu/artoolkit/ http://artoolkit.sourceforge.net/
  5. 5. How it Works
  6. 6. ARToolKit Software  ARToolKit version: 2.65 or later  Currently two license models   Open Source (GPL): ARToolKit 2.72   Commercial (ARToolWorks): ARToolKit 4.0  OS: Linux, Windows, MacOS X, iPhone/Android  Programming language: C  Related software   ARToolKit Professional: Commercial version   ARToolKitPlus: Advanced version   NyARToolkit: Java and C# version   FLARToolKit: Flash version
  7. 7. ARToolKit Family ARToolKit NFTARToolKit ARToolKit Plus JARToolKit (Java) ARToolKit (Symbian) FLARToolKit (Flash) NyToolKit - Java, C#, - Android, WM FLARManager (Flash)
  8. 8. ARToolKit Structure ARvideo.lib DirectShow  Three key libraries:   AR32.lib – ARToolKit image processing functions   ARgsub32.lib – ARToolKit graphics functions   ARvideo.lib – DirectShow video capture class
  9. 9. Additional Software  ARToolKit just provides tracking   For an AR application you’ll need more software  High level rendering library   Open VRML, Open Inventor, osgART, etc  Audio Library   Fmod, etc  Peripheral support
  10. 10. What does ARToolKit Calculate?   Position of makers in the camera coordinates  Pose of markers in the camera coordinates  Output format   3x4 matrix format to represent the transformation matrix from the marker coordinates to the camera coordinates
  11. 11. Coordinate Systems
  12. 12. From Marker To Camera  Rotation & Translation TCM : 4x4 transformation matrix from marker coord. to camera coord.
  13. 13. AR Application Development
  14. 14. An ARToolKit Application  Initialization   Load camera and pattern parameters  Main Loop   Step1. Image capture and display   Step2. Marker detection   Step3. Marker identification   Step4. Getting pose information   Step5. Object Interactions/Simulation   Step6. Display virtual objects  End Application   Camera shut down
  15. 15. Sample ARToolKit Applications Ex. 1: Simple video display Ex. 2: Detecting a marker Ex. 3: Using pattern Ex. 4: Getting a 3D information Ex. 5: Virtual object overlay
  16. 16. Ex 1: Simple Video Display  Program : sample1.c  Key points   Loop structure   Video image handling   Camera parameter handling   Window setup   Mouse and keyboard handling
  17. 17. Sample1.c – video initialization  Configure the video input vconf = <video configuration string>  Start video capture arVideoCapStart();  In init(), open the video arVideoOpen( vconf ); arVideoInqSize(&xsize, &ysize);  When finished, close the video path arVideoCapStop(); arVideoClose();
  18. 18. Changing Image Size  For input capture " vconf = “videoWidth=320,videoHeight=240"; Note – the camera must support this image size  For display argInit( &cparam, 1.5, 0, 0, 0, 0 );The second parameter means zoom ratio for display image size related to input image.
  19. 19. Graphics handling: libARgsub  Set up and clean up the graphics window void argInit( ARParam *cparam, double zoom, int fullFlag, int xwin, int ywin, int hmd_flag ); void argCleanup( void ); cparam: camera parameter zoom: zoom ratio fullFlag: 0: normal, 1: full screen mode Xwin, ywin: create small window for debug hmd_flag: 0: normal, 1: optical see-through mode
  20. 20. Sample1.c Main Functionmain()!{!!init();!!argMainLoop( mouseEvent, !! !keyEvent, mainLoop); !}!
  21. 21. Graphics handling: libARgsub  Go into the iterative cycle void argMainLoop( void (*mouseFunc)(int btn,int state,int x,int y), void (*keyFunc)(unsigned char key, int x, int y), void (*mainFunc)(void) );  Swap buffers void argSwapBuffers( void );
  22. 22. Sample1.c - mainLoop Function if( dataPtr = (ARUint8 *) arVideoGetImage()) == NULL ) { arUtilSleep(2); return; } argDrawMode2D(); argDispImage(dataPtr, 0, 0 ); arVideoCapNext(); argSwapBuffers();
  23. 23. Image capture: libARvideo  Return the pointer for captured image   ARUint8 *arVideoGetImage( void );  Pixel format and byte size are defined in config.h   #define AR_PIX_FORMAT_BGR   #define AR_PIX_SIZE 3
  24. 24. Graphics handling: libARgsub  Set the window for 2D drawing void argDrawMode2D( void );  Set the window for 3D drawing void argDrawMode3D( void ); void argDraw3dCamera( int xwin, int ywin );  Display image void argDispImage( ARUint8 *image, int xwin, int ywin );
  25. 25. Ex. 2: Detecting a Marker  Program : sample2.c  Key points   Threshold value   Important external variables   arDebug – keep thresholded image   arImage – pointer for thresholded image   arImageProcMode – use 50% image for image processing -  AR_IMAGE_PROC_IN_FULL -  AR_IMAGE_PROC_IN_HALF
  26. 26. Sample2.c – marker detection/* detect the markers in the video frame */if(arDetectMarker(dataPtr, thresh, &marker_info, &marker_num) < 0 ) { cleanup(); exit(0);}for( i = 0; i < marker_num; i++ ) { argDrawSquare(marker_info[i].vertex,0,0);}
  27. 27. Sample2.c – marker_info structure typedef struct { int area; int id; int dir; double cf; double pos[2]; double line[4][3]; double vertex[4][2]; } ARMarkerInfo; !
  28. 28. Ex. 3: Using a Pattern  Program : sample3.c  Key points   Pattern files loading   Structure of marker information -  Region features -  Pattern Id, direction -  Certainty factor   Marker identification
  29. 29. Making a pattern template  Use of utility program: mk_patt.exe  Show the pattern  Put the corner of red line segments on the left-top vertex of the marker  Pattern stored as a template in a file  1:2:1 ratio determines the pattern region used
  30. 30. Sample3.c – Pattern File Loadingint patt_id;char *patt_name = “Data/kanjiPatt”/* load pattern file */if(patt_id=arLoadPatt (patt_name) < 0){ printf ("Pattern file load error !! n"); exit(0);}
  31. 31. Checking for known patterns/* check for known patterns */ k = -1;for( i = 0; i < marker_num; i++ ) { if( marker_info[i].id == patt_id) { /* youve found a pattern */ printf("Found pattern: %d n",patt_id); if( k == -1 ) k = i; else /* make sure you have the best pattern (highest confidence factor) */ if( marker_info[k].cf < marker_info[i].cf ) k = i; }}
  32. 32. Ex. 4 – Getting 3D information  Program : sample4.c  Key points   Definition of a real marker   Transformation matrix -  Rotation component -  Translation component
  33. 33. Sample4.c – Transformation matrixdouble marker_center[2] = {0.0, 0.0};double marker_width = 80.0;double marker_trans[3][4];arGetTransMat(&marker_info[i], marker_center, marker_width, marker_trans);
  34. 34. Finding the Camera PositionThis function sets transformation matrix from marker to camera into marker_trans[3][4]." arGetTransMat(&marker_info[k], marker_center, marker_width, marker_trans);You can see the position information in the values of marker_trans[3][4]." " Xpos = marker_trans[0][3]; Ypos = marker_trans[1][3]; Zpos = marker_trans[2][3];
  35. 35. ARToolKit Coordinate Frame
  36. 36. Ex. 5- Virtual Object Display  Program : sample5.c  Key points   OpenGL parameter setting   Setup of projection matrix   Setup of modelview matrix
  37. 37. Appending your own OpenGL codeSet the camera parameters to OpenGL Projection matrix. argDrawMode3D(); argDraw3dCamera( 0, 0 );Set the transformation matrix from the marker to the camera to the OpenGL ModelView matrix. argConvGlpara(marker_trans, gl_para); glMatrixMode(GL_MODELVIEW); glLoadMatrixd( gl_para );After calling these functions, your OpenGL objects aredrawn in the real marker coordinates.
  38. 38. 3D CG Model Rendering  ARToolKit does not have a function to handle 3D CG models.  3rd party CG rendering software should be employed.   OpenVRML   OpenSceneGraph   etc
  39. 39. Loading Multiple Patterns  Sample File: LoadMulti.c   Uses object.c to load  Object Structure typedef struct { char name[256]; int id; int visible; double marker_coord[4][2]; double trans[3][4]; double marker_width; double marker_center[2]; } ObjectData_T;
  40. 40. Finding Multiple Transforms  Create object listObjectData_T *object;  Read in objects - in init( )read_ObjData( char *name, int *objectnum );  Find Transform – in mainLoop( )for( i = 0; i < objectnum; i++ ) { ..Check patterns ..Find transforms for each marker }
  41. 41. Drawing Multiple Objects  Send the object list to the draw functiondraw( object, objectnum );  Draw each object individuallyfor( i = 0; i < objectnum; i++ ) { if( object[i].visible == 0 ) continue; argConvGlpara(object[i].trans, gl_para); draw_object( object[i].id, gl_para);}
  42. 42. Limitations of ARToolKit  Partial occlusions cause tracking failure  Affected by lighting and shadows  Tracking range depends on marker size  Performance depends on number of markers   cf artTag, ARToolKitPlus  Pose accuracy depends on distance to marker  Pose accuracy depends on angle to marker
  43. 43. ARToolKit in the World  Hundreds of projects  Large research community
  44. 44. FLARToolKit  Flash AS3 Version of the ARToolKit (was ported from NyARToolkit the Java Version of the ARToolkit)  enables augmented reality in the Browser  uses Papervision3D for as 3D Engine  available at http://saqoosha.net/  dual license, GPL and commercial license
  45. 45. AR Application Components Adobe Flash Papervision 3D FLARToolkit
  46. 46. private function mainEnter(e:Event):void { /* Capture video frame*/ capture.draw(vid); /* Detect marker */ if (detector.detectMarkerLite(raster, 80) && detector.getConfidence() > 0.5) { //Get the transfomration matrix for the current marker position detector.getTransformMatrix(trans); //Translates and rotates the mainContainer so it looks right mainContainer.setTransformMatrix(trans); //Render the papervision scene renderer.render(); } }
  47. 47. FLARToolKit Examples
  48. 48. Papervision 3D  http://www.papervision3d.org/  Flash-based 3D-Engine  Supports   import of 3D Models   texturing   animation   scene graph  alternatives: Away3d, Sandy,…
  49. 49. Source Packages  „Original“ FLARToolkit (Libspark, Saqoosha) ( http://www.libspark.org/svn/as3/FLARToolKit/trunk/ )  Start-up-guides   Saqoosha (http://saqoosha.net/en/flartoolkit/start-up-guide/ )   Miko Haapoja (http://www.mikkoh.com/blog/?p=182 )  „Frameworks“   Squidder MultipleMarker – Example ( http://www.squidder.com/2009/03/06/flar-how-to-multiple-instances-of-multiple- markers/ )   FLARManager (http://words.transmote.com/wp/flarmanager/ )
  50. 50. Other Languages  NyARToolKit   http://nyatla.jp/nyartoolkit/wp/   AS3, Java, C#, Processing, Unity, etc  openFrameworks   http://www.openframeworks.cc/   https://sites.google.com/site/ofauckland/examples/8-artoolkit-example   Support for other libraries -  Kinect, Audio, Physics, etc
  51. 51. void testApp::update(){ //capture video and detect markers mov.update(); if (mov.isFrameNew()) { img.setFromPixels(mov.getPixels(), ofGetWidth(), ofGetHeight()); gray = img; tracker.setFromPixels(gray.getPixels()); }}//--------------------------------------------------------------void testApp::draw(){ //draw AR objects ofSetColor(0xffffff); mov.draw(0, 0); for (int i=0; i<tracker.markers.size(); i++) { ARMarkerInfo &m = tracker.markers[i]; tracker.loadMarkerModelViewMatrix(m); ofSetColor(255, 255, 0, 100); ofCircle(0,0,25); ofSetColor(0); ofDrawBitmapString(ofToString(m.id),0,0); }}
  52. 52. AR Tools
  53. 53. Building Compelling AR Experiences experiences applications tools Authoring components Tracking, Display Sony CSL © 2004
  54. 54. AR Authoring  Software Libraries   osgART, Studierstube, MXRToolKit  Plug-ins to existing software   DART (Macromedia Director), mARx, Unity,  Stand Alone   AMIRE, BuildAR, etc  Next Generation   iaTAR (Tangible AR)
  55. 55. mARx Plug-in  3D Studio Max Plug-in  Can model and view AR content at the same time
  56. 56. BuildAR  http://www.buildar.co.nz/  Stand alone application  Visual interface for AR model viewing application  Enables non-programmers to build AR scenes
  57. 57. Plug-ins for 3D authoring tools  Unity - Game development tool (www.unity3d.com)  Esperient Creator – General interactive 3D authoring (www.esperient.com)
  58. 58. Metaio Design  Complete commercial authoring platform   http://www.metaio.com/software/design/   Offers viewer and editor tools   Drag and drop tools   3D model import
  59. 59. Total Immersion D’Fusion Studio  Complete commercial authoring platform   http://www.t-immersion.com/   Multi-platform   Markerless tracking   Scripting   Face tracking   Finger tracking   Kinect support
  60. 60. Others  AR-Media   http://www.inglobetechnologies.com/   Google sketch-up plug-in  LinceoVR   http://linceovr.seac02.it/   AR/VR authoring package  Libraries   JARToolKit, MXRToolKit, ARLib, Goblin XNA
  61. 61. OSGART Programming Library  Integration of ARToolKit with a High-Level Rendering Engine (OpenSceneGraph) OSGART= OpenSceneGraph + ARToolKit  Supporting Geometric + Photometric Registration
  62. 62. osgART:Features  C++ (but also Python, Lua, etc).  Multiple Video Input supports:   Direct (Firewire/USB Camera), Files, Network by ARvideo, PtGrey, CVCam, VideoWrapper, etc.  Benefits of Open Scene Graph   Rendering Engine, Plug-ins, etc
  63. 63. Advanced Authoring: iaTAR (Lee 2004)  Immersive AR Authoring  Using real objects to create AR applications
  64. 64. More Information•  Mark Billinghurst –  mark.billinghurst@hitlabnz.org •  Websites –  www.hitlabnz.org

×