MediaStream processing pipelines on the Augmented Web Platform

1,602 views

Published on

An introduction to Stream processing pipelines with a focus on MediaStream pipelines. Then a deep dive into the Video/Canvas pipeline and how that can be used to process streams of image frames in real-time.

Find out more about why ArrayBuffers are the key innovation driving this evolution and how WebRTC will make this pervasive.

Published in: Sports
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,602
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • http://www.khronos.org/streaminput/
  • MediaStream processing pipelines on the Augmented Web Platform

    1. 1. https://buildAR.com - image credit It's alive! MediaStream processing pipelines on the Augmented Web Platform
    2. 2. Who's making these outlandish claims? Rob Manson @nambor CEO of MOB-labs the creators of buildAR.com Chair of the W3C's Augmented Web Community Group Invited Expert with the ISO, W3C & the Khronos Group Author of “Getting started with WebRTC” - (Packt Publishing) https://buildAR.com
    3. 3. What is the Augmented Web? https://buildAR.com
    4. 4. It's what comes after HTML5 https://buildAR.com
    5. 5. The web is evolving... 1992-2000 WWW: Page based websites key innovation: HREF 2000-2013 Web 2.0: AJAX driven social services key innovation: XHR 2013-... AWE: Sensor driven real-time web of things key innovation: ArrayBuffer http://mob-labs.com : image credit
    6. 6. https://buildAR.com
    7. 7. https://buildAR.com JSARToolkit demo
    8. 8. https://buildAR.com FAST algorithm feature detection demo
    9. 9. Test your device at: theAWEsomeWEB.com https://buildAR.com
    10. 10. So what the hell is a Stream processing pipeline? https://buildAR.com
    11. 11. https://buildAR.com Stream processing pipelines 1. Get Stream 2. Connect to a Stream pipeline A way to connect a Stream to an ArrayBuffer 3. Get ArrayBuffer 5. Populate a scene Array with Array Buffer Views 6. Move through Views to process data 7. Output events and metadata 8. Update UI and/or send requests
    12. 12. Then what is a MediaStream processing pipeline? https://buildAR.com
    13. 13. https://buildAR.com MediaStream processing pipelines 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here!
    14. 14. https://buildAR.com This is where the APIs are still evolving 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
    15. 15. https://buildAR.com This is where the Augmented Web comes to life 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
    16. 16. https://buildAR.com This is where the fun is! 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
    17. 17. #$!%@#%@#$@$@!!!!! https://buildAR.com
    18. 18. https://buildAR.com Browser's have always had a “binary barrier” The Video element stream -> <video>
    19. 19. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
    20. 20. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output
    21. 21. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output
    22. 22. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output Plus other non-MediaStream pipelines: ????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output
    23. 23. Lets take a closer look at the Video/Canvas MediaStream pipeline https://buildAR.com
    24. 24. https://buildAR.com The Video/Canvas MediaStream pipeline Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! Follow along on github: github.com/buildar/getting_started_with_webrtc
    25. 25. https://buildAR.com The Video/Canvas MediaStream pipeline Follow along on github: github.com/buildar/getting_started_with_webrtc Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all setInterval() RequestAnimationFrame() setTimeout()
    26. 26. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
    27. 27. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
    28. 28. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
    29. 29. https://buildAR.com Join the W3C's Augmented Web Community Group http://w3.org/community/ar

    ×