• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
MediaStream processing pipelines on the Augmented Web Platform
 

MediaStream processing pipelines on the Augmented Web Platform

on

  • 1,027 views

An introduction to Stream processing pipelines with a focus on MediaStream pipelines. Then a deep dive into the Video/Canvas pipeline and how that can be used to process streams of image frames in ...

An introduction to Stream processing pipelines with a focus on MediaStream pipelines. Then a deep dive into the Video/Canvas pipeline and how that can be used to process streams of image frames in real-time.

Find out more about why ArrayBuffers are the key innovation driving this evolution and how WebRTC will make this pervasive.

Statistics

Views

Total Views
1,027
Views on SlideShare
1,027
Embed Views
0

Actions

Likes
0
Downloads
7
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • http://www.khronos.org/streaminput/

MediaStream processing pipelines on the Augmented Web Platform MediaStream processing pipelines on the Augmented Web Platform Presentation Transcript

  • https://buildAR.com - image credit It's alive! MediaStream processing pipelines on the Augmented Web Platform
  • Who's making these outlandish claims? Rob Manson @nambor CEO of MOB-labs the creators of buildAR.com Chair of the W3C's Augmented Web Community Group Invited Expert with the ISO, W3C & the Khronos Group Author of “Getting started with WebRTC” - (Packt Publishing) https://buildAR.com
  • What is the Augmented Web? https://buildAR.com
  • It's what comes after HTML5 https://buildAR.com
  • The web is evolving... 1992-2000 WWW: Page based websites key innovation: HREF 2000-2013 Web 2.0: AJAX driven social services key innovation: XHR 2013-... AWE: Sensor driven real-time web of things key innovation: ArrayBuffer http://mob-labs.com : image credit
  • https://buildAR.com
  • https://buildAR.com JSARToolkit demo
  • https://buildAR.com FAST algorithm feature detection demo
  • Test your device at: theAWEsomeWEB.com https://buildAR.com
  • So what the hell is a Stream processing pipeline? https://buildAR.com
  • https://buildAR.com Stream processing pipelines 1. Get Stream 2. Connect to a Stream pipeline A way to connect a Stream to an ArrayBuffer 3. Get ArrayBuffer 5. Populate a scene Array with Array Buffer Views 6. Move through Views to process data 7. Output events and metadata 8. Update UI and/or send requests
  • Then what is a MediaStream processing pipeline? https://buildAR.com
  • https://buildAR.com MediaStream processing pipelines 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here!
  • https://buildAR.com This is where the APIs are still evolving 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  • https://buildAR.com This is where the Augmented Web comes to life 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  • https://buildAR.com This is where the fun is! 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  • #$!%@#%@#$@$@!!!!! https://buildAR.com
  • https://buildAR.com Browser's have always had a “binary barrier” The Video element stream -> <video>
  • https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
  • https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output
  • https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output
  • https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output Plus other non-MediaStream pipelines: ????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output
  • Lets take a closer look at the Video/Canvas MediaStream pipeline https://buildAR.com
  • https://buildAR.com The Video/Canvas MediaStream pipeline Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! Follow along on github: github.com/buildar/getting_started_with_webrtc
  • https://buildAR.com The Video/Canvas MediaStream pipeline Follow along on github: github.com/buildar/getting_started_with_webrtc Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all setInterval() RequestAnimationFrame() setTimeout()
  • https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  • https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  • https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  • https://buildAR.com Join the W3C's Augmented Web Community Group http://w3.org/community/ar