Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Augmented Reality
in WebRTC Browser
- Altanai Bisht
tara181989@gmail.com
http://altanaitelecom.wordpress.com
Who am I ?
★ Btech IT degree from Anna University , Chennai
★ 4 years in telecom industry
★ Frequent contributor to open s...
Augmented reality (AR) is viewing a real-world
environment with elements that are
supplemented by computer-generated senso...
How is it diff. from Virtual Reality ?
Virtual Reality Augmented Reality
replaces the real world with simulated
one
blendi...
Methods for rendering Augmented Reality
Computer Vision
Object Recognition
Eye Tracking
Face Detection and substitution
Em...
Web :
WebRTC getusermedia
Web Speech API
WebGL
css
svg
HTML5 canvas
sensor API
Components of end -to-end web based
Augment...
WebRTC
7
8
What is WebRTC ?
● Web based Real Time communications
● Definition for browser's media stream and data
● Awaiting more s...
WebRTC implements 3 APIs:
getUserMedia
RTCPeerConnection
RTCDataChannel
9
API Stack
Code snippets for WebRTC
10
1.To begin with WebRTC we first need to validate that the browser has permission to access the...
AppRTC
11
Simple WebRTC API maintained by google
https://apprtc.appspot.com
TFX : collaboration and communication with webrtc
12
TFX : collaboration and communication with webrtc
13
14
15
What is WebGL?
● Web Graphics Library
● JavaScript API for rendering interactive 2D and 3D computer graphics in
browser...
To get started you should know about :
● GLSL, the shading language used by OpenGL and WebGL
● Matrix computation to set u...
Building 3D graphics with
Javascript
17
JS Libraries for 3D graphics
CCV
website : http://libccv.org/
SourceCode : https://github.com/liuliu/ccv
Demo:
Three.js
we...
Features
● MIT license
● javascript 3D engine ie ( WebGL
+ more)
● started a year ago
● under active development
● no depe...
20
1. Spinning Colored Cube
2. Shaded Material on Sphere
3. Complicated materials like a bun
4. 3D on webcam input as text...
1. Get three.js from : http://threejs.org/build/three.min.js
2. Make a empty HTML5 page and import the script + basic styl...
3. Scene
var scene = new THREE.Scene();
4. Camera
Camera types in three.js are CubeCamera , OrthographicCamera, Perspectiv...
6. BoxGeometry object contains all the points (vertices) and fill (faces) of the cube.
var geometry = new THREE.BoxGeometr...
10. Create a loop to render something on the screen
function render() {
requestAnimationFrame( render );
renderer.render( ...
output:
25
2. Shaded Material on Sphere
1. create a empty page and import three.min.js and jquery
<html>
<head>
<title>Shaded Materia...
27
2. Repeat the same steps at in previous example
var scene = new THREE.Scene();
var camera = new THREE.PerspectiveCamera...
5. Light
Create light , set its position and add it to scene as well . Light can be point light , spot light , directional...
output:
29
3. Complex objects like
Torusknot
1.Same as before make scene , camera and renderer
scene = new THREE.Scene();
camera = ne...
31
3. add Torusknotgeometry with radius, tube, radialSegments, tubularSegments, arc
var geometry = new THREE.TorusKnotGeom...
output:
32
4. 3D with webcam texture
Display the video as a plane which can be viewed from various angles in a given background lands...
34
5. Make the FLOOR with an image texture
var floorTexture = new THREE.ImageUtils.loadTexture( 'imageURL.jpg' );
floorTex...
35
7.Add video Image Context and Texture.
video = document.getElementById( 'monitor' );
videoImage = document.getElementBy...
36
8. Set camera position
camera.position.set(0,150,300);
camera.lookAt(movieScreen.position);
9. Define the render functi...
output:
37
4. Button touch detector
This example shows the process of triggering Web page activity by detecting motion using differen...
39
4. Define blend
// get current webcam image data
var sourceData = videoContext.getImageData(0, 0, width, height);
// cr...
output Differential :
40
output:
Other WebRTC WebGL 3D JS
applications
41
karenLabs
42
http://www.karenlabs.com/music/#.VUwvYFE4bK4
MotionEmotion
43
http://motionemotion.herokuapp.com/
Mosh.io
44
http://www.karenlabs.com/music/#.VUwvYFE4bK4
DeerHeaven
45
http://hayleejung.com/deerheaven/
motion and gesture detector using webcam inout
Cubeslam
46
https://www.cubeslam.com/mgeumt
WebRTC based 2 party game
47
Thats all folks !
Upcoming SlideShare
Loading in …5
×

Augmented reality in web rtc browser

3,424 views

Published on

Augmented Reality and WebRTC .
WebGL in browsers
threejs for 3D graphics
code and steps of making cube , sphere , torusknot
wecam texture to 3D plane
motion detection using differential analysis
more samples of AR applications

Published in: Technology
  • Sex in your area is here: ❶❶❶ http://bit.ly/2Q98JRS ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: ❤❤❤ http://bit.ly/2Q98JRS ❤❤❤
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Augmented reality in web rtc browser

  1. 1. Augmented Reality in WebRTC Browser - Altanai Bisht tara181989@gmail.com http://altanaitelecom.wordpress.com
  2. 2. Who am I ? ★ Btech IT degree from Anna University , Chennai ★ 4 years in telecom industry ★ Frequent contributor to open source software ★ Write at : http://altanaitelecom.wordpress.com ★ Author of book “ WebRTC Integrator’s Guide “ 2 Altanai Bisht
  3. 3. Augmented reality (AR) is viewing a real-world environment with elements that are supplemented by computer-generated sensory inputs such as sound, video, graphics , location etc. What is Augmented Reality ? 3
  4. 4. How is it diff. from Virtual Reality ? Virtual Reality Augmented Reality replaces the real world with simulated one blending of virtual reality and real life user is isolated from real life user interacts with real world through digital overlays Oculus Rift Kinect Google glass Holo Lens 4
  5. 5. Methods for rendering Augmented Reality Computer Vision Object Recognition Eye Tracking Face Detection and substitution Emotion and gesture picker Edge Detection 5
  6. 6. Web : WebRTC getusermedia Web Speech API WebGL css svg HTML5 canvas sensor API Components of end -to-end web based Augmented Reality solution 6 H/w : Graphics driver microphone and camera sensors 3D : Geometry and Math Utilities 3D Model Loaders and models Lights, Materials,Shaders, Particles, Animation
  7. 7. WebRTC 7
  8. 8. 8 What is WebRTC ? ● Web based Real Time communications ● Definition for browser's media stream and data ● Awaiting more standardization , on a API level at the W3C and at the protocol level at the IETF. ● Enable browser to browser applications for voice calling, video chat and P2P file sharing without plugins. ● Enables web browsers with Real-Time Communications (RTC) capabilities ● MIT : Free, open project
  9. 9. WebRTC implements 3 APIs: getUserMedia RTCPeerConnection RTCDataChannel 9 API Stack
  10. 10. Code snippets for WebRTC 10 1.To begin with WebRTC we first need to validate that the browser has permission to access the webcam. <video id="webcam" autoplay width="640" height="480"></video> 2. Find out if the user's browser can use the getUserMedia API. function hasGetUserMedia() { return !!(navigator.webkitGetUserMedia); } 3. Get the stream from the user's webcam. var video = $('#webcam')[0]; if (navigator.webkitGetUserMedia) { navigator.webkitGetUserMedia( {audio:true, video:true}, function(stream) { video.src = window.webkitURL.createObjectURL(stream); }, function(e) { alert('Webcam error!', e); } ); }
  11. 11. AppRTC 11 Simple WebRTC API maintained by google https://apprtc.appspot.com
  12. 12. TFX : collaboration and communication with webrtc 12
  13. 13. TFX : collaboration and communication with webrtc 13
  14. 14. 14
  15. 15. 15 What is WebGL? ● Web Graphics Library ● JavaScript API for rendering interactive 2D and 3D computer graphics in browser ● no plugins ● uses GPU ( Graphics Processing Unit ) acceleration ● can mix with other HTML elements ● uses the HTML5 canvas element and is accessed using Document Object Model interfaces ● cross platform , works on all major Desktop and mobile browsers
  16. 16. To get started you should know about : ● GLSL, the shading language used by OpenGL and WebGL ● Matrix computation to set up transformations ● Vertex buffers to hold data about vertex positions, normals, colors, and textures ● matrix math to animate shapes Cleary WebGL is bit tough given the amount of careful coding , mapping and shading it requires . Proceeding to some JS libraries that can make 3D easy for us . 16 WebGL development
  17. 17. Building 3D graphics with Javascript 17
  18. 18. JS Libraries for 3D graphics CCV website : http://libccv.org/ SourceCode : https://github.com/liuliu/ccv Demo: Three.js website : http://threejs.org/ SourceCode : https://github.com/mrdoob/three.js/ Demo: http://www.davidscottlyons.com/threejs/ Awe.js Website : https://buildar.com/awe/tutorials/intro_to_awe. js/index.html# SourceCode : https://github.com/buildar/awe.js 18 ArcuCO SourceCode: https://github.com/jcmellado/js-aruco Potree website: http://potree.org/wp/ SourceCode: https://github.com/potree Demo: http://potree.org/wp/demo/ Karenpeng emotion & gesture-based arpeggiator and synthesizer website: SourceCode : https://github.com/karenpeng/motionEmotion Demo: http://motionemotion.herokuapp.com/
  19. 19. Features ● MIT license ● javascript 3D engine ie ( WebGL + more) ● started a year ago ● under active development ● no dependencies or installation 19 1. Set the scene , camera , renderer 2. Set the Mesh with Geometry & materials 3. Lights 4. Object properties 5. Add to scene , set camera position 6. Render with scene and camera 7. Animation Steps Three.js
  20. 20. 20 1. Spinning Colored Cube 2. Shaded Material on Sphere 3. Complicated materials like a bun 4. 3D on webcam input as texture 5. Motion Detection on webcam input Lets Code !
  21. 21. 1. Get three.js from : http://threejs.org/build/three.min.js 2. Make a empty HTML5 page and import the script + basic styling of page <html> <head> <title>Spinning colored Cube</title> <style> body { margin: 0; } canvas { width: 100%; height: 100% } </style> </head> <body> <script src="js/three.min.js"></script> <script>// Our Javascript will go here. </script> </body> </html> 1. Spinning Colored Cube 21
  22. 22. 3. Scene var scene = new THREE.Scene(); 4. Camera Camera types in three.js are CubeCamera , OrthographicCamera, PerspectiveCamera. We are using Perspective camera here . Attributes are field of view , aspect ratio , near and far clipping plane. var camera = new THREE.PerspectiveCamera( 75, window.innerWidth / window.innerHeight, 0.1, 1000 ); 5. Renderer Renderer uses a <canvas> element to display the scene to us. var renderer = new THREE.WebGLRenderer(); renderer.setSize( window.innerWidth, window.innerHeight ); document.body.appendChild( renderer.domElement ); 22
  23. 23. 6. BoxGeometry object contains all the points (vertices) and fill (faces) of the cube. var geometry = new THREE.BoxGeometry( 1, 1, 1 ); 7. Material threejs has materials like - LineBasicMaterial , MeshBasicMaterial , MeshPhongMaterial , MeshLambertMaterial These have their properties like -id, name, color , opacity , transparent etc. Use MeshBasicMaterial and color attribute of 0x00ff00, which is green. var material = new THREE.MeshBasicMaterial( { color: 0x00ff00 } ); 8. Mesh A mesh is an object that takes a geometry, and applies a material to it, which we then can insert to our scene, and move freely around. var cube = new THREE.Mesh( geometry, material ); 9. By default, when we call scene.add(), the thing we add will be added to the coordinates (0,0,0). This would cause both the camera and the cube to be inside each other. To avoid this, we simply move the camera out a bit. scene.add( cube ); camera.position.z = 5; 23
  24. 24. 10. Create a loop to render something on the screen function render() { requestAnimationFrame( render ); renderer.render( scene, camera ); } render(); This will create a loop that causes the renderer to draw the scene 60 times per second. 11. Animating the cube This will be run every frame (60 times per second), and give the cube a nice rotation animation cube.rotation.x += 0.1; cube.rotation.y += 0.1; 24
  25. 25. output: 25
  26. 26. 2. Shaded Material on Sphere 1. create a empty page and import three.min.js and jquery <html> <head> <title>Shaded Material on Sphere </title> <style> body { margin: 0; } canvas { width: 100%; height: 100% } </style> <script src="js/jquery.min.js"></script> <script src="js/three.min.js"></script> <script>// Our Javascript will go here.</script> </head> <body> <div id="container"></div> </body> </html> 26
  27. 27. 27 2. Repeat the same steps at in previous example var scene = new THREE.Scene(); var camera = new THREE.PerspectiveCamera(45, 600/600 , 0.1, 10000); var renderer = new THREE.WebGLRenderer(); renderer.setSize(600 , 600 ); $container.append(renderer.domElement); scene.add(camera); camera.position.z = 300; // the camera starts at 0,0,0 so pull it back 3. Create the sphere's material as MeshLambertMaterial MeshLambertMaterial is non-shiny (Lambertian) surfaces, evaluated per vertex. Set the color to red . var sphereMaterial = new THREE.MeshLambertMaterial( { color: 0xCC0000 }); 4. create a new mesh with sphere geometry ( radius, segments, rings) and add to scene var sphere = new THREE.Mesh( new THREE.SphereGeometry( 50, 16, 16 ), sphereMaterial); scene.add(sphere);
  28. 28. 5. Light Create light , set its position and add it to scene as well . Light can be point light , spot light , directional light . var pointLight = new THREE.PointLight(0xFFFFFF); pointLight.position.x = 10; pointLight.position.y = 50; pointLight.position.z = 130; scene.add(pointLight); 6. Render the whole thing renderer.render(scene, camera); 28
  29. 29. output: 29
  30. 30. 3. Complex objects like Torusknot 1.Same as before make scene , camera and renderer scene = new THREE.Scene(); camera = new THREE.PerspectiveCamera(125, window.innerWidth / window.innerHeight, 1, 500); camera.position.set(0, 0, 100); camera.lookAt(new THREE.Vector3(0, 0, 0)); var renderer = new THREE.WebGLRenderer(); renderer.setSize( window.innerWidth, window.innerHeight ); document.body.appendChild( renderer.domElement ); 2. Add the lighting var light = new THREE.PointLight(0xffffff); light.position.set(0, 250, 0); scene.add(light); var ambientLight = new THREE.AmbientLight(0x111111); scene.add(ambientLight); 30
  31. 31. 31 3. add Torusknotgeometry with radius, tube, radialSegments, tubularSegments, arc var geometry = new THREE.TorusKnotGeometry( 8, 2, 100, 16, 4, 3 ); var material = new THREE.MeshLambertMaterial( { color: 0x2022ff } ); var torusKnot = new THREE.Mesh( geometry, material ); torusKnot.position.set(3, 3, 3); scene.add( torusKnot ); camera.position.z =25; 4.do the animation and render on screen var render = function () { requestAnimationFrame( render ); torusKnot.rotation.x += 0.01; torusKnot.rotation.y += 0.01; renderer.render(scene, camera); }; render();
  32. 32. output: 32
  33. 33. 4. 3D with webcam texture Display the video as a plane which can be viewed from various angles in a given background landscape. 1.Use code from slide 10 to get user’s webcam input through getUserMedia 2. Make a Screen , camera and renderer as previously described 3. Give orbital CONTROLS for viewing the media plane from all angles controls = new THREE.OrbitControls( camera, renderer.domElement ); 4. Add point LIGHT to scene 33
  34. 34. 34 5. Make the FLOOR with an image texture var floorTexture = new THREE.ImageUtils.loadTexture( 'imageURL.jpg' ); floorTexture.wrapS = floorTexture.wrapT = THREE.RepeatWrapping; floorTexture.repeat.set( 10, 10 ); var floorMaterial = new THREE.MeshBasicMaterial({map: floorTexture, side: THREE.DoubleSide}); var floorGeometry = new THREE.PlaneGeometry(1000, 1000, 10, 10); var floor = new THREE.Mesh(floorGeometry, floorMaterial); floor.position.y = -0.5; floor.rotation.x = Math.PI / 2; scene.add(floor); 6. Add Fog scene.fog = new THREE.FogExp2( 0x9999ff, 0.00025 );
  35. 35. 35 7.Add video Image Context and Texture. video = document.getElementById( 'monitor' ); videoImage = document.getElementById( 'videoImage' ); videoImageContext = videoImage.getContext( '2d' ); videoImageContext.fillStyle = '#000000'; videoImageContext.fillRect( 0, 0, videoImage.width, videoImage.height ); videoTexture = new THREE.Texture( videoImage ); videoTexture.minFilter = THREE.LinearFilter; videoTexture.magFilter = THREE.LinearFilter; var movieMaterial=new THREE.MeshBasicMaterial({map:videoTexture,overdraw:true,side:THREE.DoubleSide}); var movieGeometry = new THREE.PlaneGeometry( 100, 100, 1, 1 ); var movieScreen = new THREE.Mesh( movieGeometry, movieMaterial ); movieScreen.position.set(0,50,0); scene.add(movieScreen);
  36. 36. 36 8. Set camera position camera.position.set(0,150,300); camera.lookAt(movieScreen.position); 9. Define the render function videoImageContext.drawImage( video, 0, 0, videoImage.width, videoImage.height ); renderer.render( scene, camera ); 10. Animation requestAnimationFrame( animate ); render();
  37. 37. output: 37
  38. 38. 4. Button touch detector This example shows the process of triggering Web page activity by detecting motion using difference accuracy 1. Follow the same steps as in previous example 2. Define as many buttons var buttons = []; var button1 = new Image(); button1.src ="https://stemkoski.github.io/Three.js/images/SquareRed.png"; var buttonData1 = { name:"red", image:button1, x:320 - 96 - 30, y:10, w:32, h:32 }; buttons.push( buttonData1 ); 3. make 3 layers of canvas for video , buttons and blended repectively . 38
  39. 39. 39 4. Define blend // get current webcam image data var sourceData = videoContext.getImageData(0, 0, width, height); // create an image if the previous image doesn’t exist if (!lastImageData) lastImageData = videoContext.getImageData(0, 0, width, height); // create a ImageData instance to receive the blended result var blendedData = videoContext.createImageData(width, height); // blend the 2 images differenceAccuracy(blendedData.data, sourceData.data, lastImageData.data); // draw the result in a canvas blendContext.putImageData(blendedData, 0, 0); // store the current webcam image lastImageData = sourceData; 5. Use mathematical logic for difference accuracy 6. The motion areas will be highlighted with white while no motion areas will become black. 7. check if white portion overlaps with buttons .
  40. 40. output Differential : 40 output:
  41. 41. Other WebRTC WebGL 3D JS applications 41
  42. 42. karenLabs 42 http://www.karenlabs.com/music/#.VUwvYFE4bK4
  43. 43. MotionEmotion 43 http://motionemotion.herokuapp.com/
  44. 44. Mosh.io 44 http://www.karenlabs.com/music/#.VUwvYFE4bK4
  45. 45. DeerHeaven 45 http://hayleejung.com/deerheaven/ motion and gesture detector using webcam inout
  46. 46. Cubeslam 46 https://www.cubeslam.com/mgeumt WebRTC based 2 party game
  47. 47. 47 Thats all folks !

×