Natural Feature Tracking in JavaScript


Published on

The poster was presented during the IEEE Virtual Reality conference (IEEE VR 2012) in Orange County, California, USA on March 4-8, 2012. IEEE VR is the premier conference for virtual reality, including work on augmented reality, mixed reality, and 3D user interfaces.

The poster presenting an efficient natural feature tracking pipeline solely implemented in JavaScript. It is embedded in a web technologybased Augmented Reality system running plugin-free in web browsers. The evaluation shows that real-time framerates on desktop computers and while on smartphones interactive framerates are achieved.

For more info about the poster, please follow this link

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Natural Feature Tracking in JavaScript

  1. 1. Natural Feature Tracking in JavaScript Christoph Oberhofer+, Jens Grubert+, Gerhard Reitmayr +INTRODUCTIONWe present an efficient natural feature tracking pipeline solely implemented in JavaScript. It is embedded in a web technology-based Augmented Reality system runningplugin-free in web browsers. An evaluation shows that real-time frame rates on a desktop computer and interactive frame rates on a smartphones are achieved.MOTIVATIONPortability HTML5Reliance on web technologies enables HTML5 provides the building blocks for realizingplatform independence. plugin-free Augmented Reality on the web.As we employ standardized web technologies our system can HTML5 elements and related frameworks (WebGL, WebRTC)run on most platforms that provide an HTML5 enabled browser. enable the creation of a complete Augmented Reality pipeline including natural feature tracking.IMPLEMENTATION Augmented Reality Pipeline in HTML5 Two-Stage Tracking Approach Camera Access DetectionWe employ the MediaStram API alongwith the <video> and <canvas > elements <video> <canvas> pose getUserMediato access the video stream of a camera. FAST [2] BRIEF [1] RANSAC multi-level keypoint detection keypoint description + matching outlier removal estimationA two-stage pipeline written entirely inJavaScript efficiently computes the poseof the camera (similar to [3]). When the poseinitial pose is detected it is incrementally 3D Rendering Computer Vision Trackingupdated during the tracking stage. Image Level n Image Level 0 … function initializationPhase(){ grabber.grab(); NCC-based NCC-based imageDataContainer.updateData();Finally, the pose is used to overlay 3D fastDetector.detect(imageDataContainer, fastThreshold, fastPoints); pose pose fastDetector.nonmax_suppression(fastPoints, keyPoints); brief.getDescriptorsForLevel(imageDataContainer, keyPoints); keypoint keypoint var correspondences = brief.match(template_keypointSet, keyPoints); CVUtils.undistortPoints(correspondences.right, K, distCoeffs);graphics with WebGL over the video. refinement refinement var ransac = new RANSAC(correspondences, config.detection.ransac); inlierSet = ransac.execute(); tracking tracking if(inlierSet == null || inlierSet.length < config.detection.ransac.minNumOfInlier){ poseUpdated(TrackResult.FAILURE); return; } var H = CHOLESKY.computeHomography(inlierSet); inlierSet = new Array(); ransac.computeInlierSet(inlierSet, H.toMat()); H = CHOLESKY.computeHomography(inlierSet); computePose(H, inlierSet); // Switch to tracking phase poseUpdated(TrackResult.GOOD); currentPhase = Phase.TRACKING; document.getElementById("numFeatures").innerHTML = inlierSet.length; }EVALUATION Detection Phase PC Detection Phase PC vs. Mobile Chrome Canary 17 Tracking Phase PC Tracking Phase PC vs. Mobile Chrome Canary 17 Firefox 8 (PC) Firefox 8 (PC) Opera Next 12 Opera Next 12 Firefox Nightly 11.0a1System Firefox 8 Firefox 8 (Android) Firefox Nightly 11.0a1 Firefox 8 (Android)PC: Core2Duo 3GHZ, 4GB RAM Google Native Client Firefox 8Mobile: Samsung Galaxy SII Adobe Alchemy Time [ms] 0 100 200 300 400 500 600 Time [ms] 0 5 10 15 20 25 30 35 40 45 50 Time [ms] 0 10 20 30 40 50 60 70 80 90 100 Time [ms] 0 20 40 60 80 100 120 140 160 keyframe track 2 pose 2 track 1 pose 1 track 0 pose 0 keyframe tracking 1 pose 1 tracking 0 pose 0 integral fast descriptor match ransac pose integral fast descriptor match ransac pose 320x240px | FAST: 250 KP | BRIEF: 128 bit | RANSAC: min 30 inlier | max 300 iterations 320x240px | levels: 3 (120, 100, 70 KPs) | search radius: 2, 2, 4 px | patch size: 8x8 px 320x240px | levels: 2 (60, 40 KPs) | search radius: 2, 3 px | patch size: 8x8 px In the detection phase the fastest web browser (Firefox 8) is 1.6 times faster than the slowest one (Opera 12). In the tracking phase the fastest web browser (Chrome Canary) is 2.5 times faster than the slowest one (Firefox 8). An implementation in Google Native Client is 3.7 times faster and in Adobe Alchemy still 2 times faster. Compared to the PC the system runs 5.5 times slower in a web browser on a smartphone. Compared to the PC the system runs 6 times slower in a web browser on a smartphone. Overall the detection can be carried out with 11 frames per second (fps) on a PC but only with 2 fps on a smartphone. Once initialized the system can track with up to 50 fps on a PC and 9 fps on a smartphone.REFERENCES [1] M. Calonder, V. Lepetit, C. Strecha, and P. Fua. BRIEF: Binary Robust Independent Elementary Features. In ECCV 2010, pages 778-792. +Graz [2] E. Rosten and T. Drummond. Machine learning for high-speed corner detection. In ECCV 2006, pages 430-443. University of Technology, Austria |[ grubert| reitmayr] | [3] D. Wagner, G. Reitmayr, A. Mulloni, T. Drummond, and D. Schmalstieg. Real-Time Detection and Tracking for Augmented Reality on Mobile Phones. In IEEE TVCG, 99(1), pages 355-368, November 2010. This work is made possible by the Austrian National Research Funding Agency FFG in the SmartReality project.