Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
https://buildAR.com - image credit
It's alive!
MediaStream processing pipelines
on the Augmented Web Platform
Who's making these outlandish claims?
Rob Manson @nambor
CEO of MOB-labs the creators of buildAR.com
Chair of the W3C's Au...
What is the Augmented Web?
https://buildAR.com
It's what comes after HTML5
https://buildAR.com
The web is evolving...
1992-2000
WWW: Page based websites
key innovation: HREF
2000-2013
Web 2.0: AJAX driven social servi...
https://buildAR.com
https://buildAR.com
JSARToolkit demo
https://buildAR.com
FAST algorithm feature detection demo
Test your device at: theAWEsomeWEB.com
https://buildAR.com
So what the hell is a
Stream processing pipeline?
https://buildAR.com
https://buildAR.com
Stream processing pipelines
1. Get Stream
2. Connect to a Stream pipeline
A way to connect a Stream to...
Then what is a
MediaStream processing pipeline?
https://buildAR.com
https://buildAR.com
MediaStream processing pipelines
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection...
https://buildAR.com
This is where the APIs are still evolving
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerC...
https://buildAR.com
This is where the Augmented Web comes to life
1. Get MediaStream
MediaStream from getUserMedia or RTCP...
https://buildAR.com
This is where the fun is!
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Con...
#$!%@#%@#$@$@!!!!!
https://buildAR.com
https://buildAR.com
Browser's have always had a “binary barrier”
The Video element
stream -> <video>
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> ...
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> ...
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> ...
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> ...
Lets take a closer look at the
Video/Canvas MediaStream pipeline
https://buildAR.com
https://buildAR.com
The Video/Canvas MediaStream pipeline
Access the camera (or some other stream) source
a. getUserMedia(...
https://buildAR.com
The Video/Canvas MediaStream pipeline
Follow along on github: github.com/buildar/getting_started_with_...
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parall...
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parall...
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parall...
https://buildAR.com
Join the W3C's
Augmented Web Community Group
http://w3.org/community/ar
Upcoming SlideShare
Loading in …5
×

MediaStream processing pipelines on the Augmented Web Platform

1,689 views

Published on

An introduction to Stream processing pipelines with a focus on MediaStream pipelines. Then a deep dive into the Video/Canvas pipeline and how that can be used to process streams of image frames in real-time.

Find out more about why ArrayBuffers are the key innovation driving this evolution and how WebRTC will make this pervasive.

Published in: Sports
  • Be the first to comment

  • Be the first to like this

MediaStream processing pipelines on the Augmented Web Platform

  1. 1. https://buildAR.com - image credit It's alive! MediaStream processing pipelines on the Augmented Web Platform
  2. 2. Who's making these outlandish claims? Rob Manson @nambor CEO of MOB-labs the creators of buildAR.com Chair of the W3C's Augmented Web Community Group Invited Expert with the ISO, W3C & the Khronos Group Author of “Getting started with WebRTC” - (Packt Publishing) https://buildAR.com
  3. 3. What is the Augmented Web? https://buildAR.com
  4. 4. It's what comes after HTML5 https://buildAR.com
  5. 5. The web is evolving... 1992-2000 WWW: Page based websites key innovation: HREF 2000-2013 Web 2.0: AJAX driven social services key innovation: XHR 2013-... AWE: Sensor driven real-time web of things key innovation: ArrayBuffer http://mob-labs.com : image credit
  6. 6. https://buildAR.com
  7. 7. https://buildAR.com JSARToolkit demo
  8. 8. https://buildAR.com FAST algorithm feature detection demo
  9. 9. Test your device at: theAWEsomeWEB.com https://buildAR.com
  10. 10. So what the hell is a Stream processing pipeline? https://buildAR.com
  11. 11. https://buildAR.com Stream processing pipelines 1. Get Stream 2. Connect to a Stream pipeline A way to connect a Stream to an ArrayBuffer 3. Get ArrayBuffer 5. Populate a scene Array with Array Buffer Views 6. Move through Views to process data 7. Output events and metadata 8. Update UI and/or send requests
  12. 12. Then what is a MediaStream processing pipeline? https://buildAR.com
  13. 13. https://buildAR.com MediaStream processing pipelines 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here!
  14. 14. https://buildAR.com This is where the APIs are still evolving 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  15. 15. https://buildAR.com This is where the Augmented Web comes to life 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  16. 16. https://buildAR.com This is where the fun is! 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  17. 17. #$!%@#%@#$@$@!!!!! https://buildAR.com
  18. 18. https://buildAR.com Browser's have always had a “binary barrier” The Video element stream -> <video>
  19. 19. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
  20. 20. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output
  21. 21. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output
  22. 22. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output Plus other non-MediaStream pipelines: ????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output
  23. 23. Lets take a closer look at the Video/Canvas MediaStream pipeline https://buildAR.com
  24. 24. https://buildAR.com The Video/Canvas MediaStream pipeline Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! Follow along on github: github.com/buildar/getting_started_with_webrtc
  25. 25. https://buildAR.com The Video/Canvas MediaStream pipeline Follow along on github: github.com/buildar/getting_started_with_webrtc Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all setInterval() RequestAnimationFrame() setTimeout()
  26. 26. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  27. 27. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  28. 28. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  29. 29. https://buildAR.com Join the W3C's Augmented Web Community Group http://w3.org/community/ar

×