An introduction to Stream processing pipelines with a focus on MediaStream pipelines. Then a deep dive into the Video/Canvas pipeline and how that can be used to process streams of image frames in real-time.
Find out more about why ArrayBuffers are the key innovation driving this evolution and how WebRTC will make this pervasive.
NCAA Tournament Free Pick, March 22 From Top March Madness Sports Handicapper
MediaStream processing pipelines on the Augmented Web Platform
1. https://buildAR.com - image credit
It's alive!
MediaStream processing pipelines
on the Augmented Web Platform
2. Who's making these outlandish claims?
Rob Manson @nambor
CEO of MOB-labs the creators of buildAR.com
Chair of the W3C's Augmented Web Community Group
Invited Expert with the ISO, W3C & the Khronos Group
Author of “Getting started with WebRTC” - (Packt Publishing)
https://buildAR.com
5. The web is evolving...
1992-2000
WWW: Page based websites
key innovation: HREF
2000-2013
Web 2.0: AJAX driven social services
key innovation: XHR
2013-...
AWE: Sensor driven real-time web of things
key innovation: ArrayBuffer
http://mob-labs.com : image credit
10. So what the hell is a
Stream processing pipeline?
https://buildAR.com
11. https://buildAR.com
Stream processing pipelines
1. Get Stream
2. Connect to a Stream pipeline
A way to connect a Stream to an ArrayBuffer
3. Get ArrayBuffer
5. Populate a scene Array with Array Buffer Views
6. Move through Views to process data
7. Output events and metadata
8. Update UI and/or send requests
12. Then what is a
MediaStream processing pipeline?
https://buildAR.com
13. https://buildAR.com
MediaStream processing pipelines
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
14. https://buildAR.com
This is where the APIs are still evolving
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
15. https://buildAR.com
This is where the Augmented Web comes to life
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
16. https://buildAR.com
This is where the fun is!
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
19. https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
20. https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
21. https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output
22. https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output
Plus other non-MediaStream pipelines:
????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output
23. Lets take a closer look at the
Video/Canvas MediaStream pipeline
https://buildAR.com
24. https://buildAR.com
The Video/Canvas MediaStream pipeline
Access the camera (or some other stream) source
a. getUserMedia()
Setup <video> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“video”) (no need to appendChild())
Pipe camera stream into <video>
a. video.src = stream
Setup <canvas> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“canvas”) then appendChild()
Get 2D drawing context
a. canvas.getContext('2d');
Draw <video> frame onto <canvas>
a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixels
a. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cycles
a. fore each(var obj in view) { … }
NOTE: Integrate other streams & sensor data here
Render results
a. using HTML/JS/CSS
b. using another <canvas> and drawImage()
c. using WebGL
d. a combination of all
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
5. Populate a scene with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
Follow along on github: github.com/buildar/getting_started_with_webrtc
25. https://buildAR.com
The Video/Canvas MediaStream pipeline
Follow along on github: github.com/buildar/getting_started_with_webrtc
Access the camera (or some other stream) source
a. getUserMedia()
Setup <video> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“video”) (no need to appendChild())
Pipe camera stream into <video>
a. video.src = stream
Setup <canvas> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“canvas”) then appendChild()
Get 2D drawing context
a. canvas.getContext('2d');
Draw <video> frame onto <canvas>
a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixels
a. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cycles
a. fore each(var obj in view) { … }
NOTE: Integrate other streams & sensor data here
Render results
a. using HTML/JS/CSS
b. using another <canvas> and drawImage()
c. using WebGL
d. a combination of all
setInterval()
RequestAnimationFrame()
setTimeout()
26. https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
27. https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
28. https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)