SlideShare a Scribd company logo
1 of 29
Download to read offline
https://buildAR.com - image credit
It's alive!
MediaStream processing pipelines
on the Augmented Web Platform
Who's making these outlandish claims?
Rob Manson @nambor
CEO of MOB-labs the creators of buildAR.com
Chair of the W3C's Augmented Web Community Group
Invited Expert with the ISO, W3C & the Khronos Group
Author of “Getting started with WebRTC” - (Packt Publishing)
https://buildAR.com
What is the Augmented Web?
https://buildAR.com
It's what comes after HTML5
https://buildAR.com
The web is evolving...
1992-2000
WWW: Page based websites
key innovation: HREF
2000-2013
Web 2.0: AJAX driven social services
key innovation: XHR
2013-...
AWE: Sensor driven real-time web of things
key innovation: ArrayBuffer
http://mob-labs.com : image credit
https://buildAR.com
https://buildAR.com
JSARToolkit demo
https://buildAR.com
FAST algorithm feature detection demo
Test your device at: theAWEsomeWEB.com
https://buildAR.com
So what the hell is a
Stream processing pipeline?
https://buildAR.com
https://buildAR.com
Stream processing pipelines
1. Get Stream
2. Connect to a Stream pipeline
A way to connect a Stream to an ArrayBuffer
3. Get ArrayBuffer
5. Populate a scene Array with Array Buffer Views
6. Move through Views to process data
7. Output events and metadata
8. Update UI and/or send requests
Then what is a
MediaStream processing pipeline?
https://buildAR.com
https://buildAR.com
MediaStream processing pipelines
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
https://buildAR.com
This is where the APIs are still evolving
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
https://buildAR.com
This is where the Augmented Web comes to life
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
https://buildAR.com
This is where the fun is!
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
#$!%@#%@#$@$@!!!!!
https://buildAR.com
https://buildAR.com
Browser's have always had a “binary barrier”
The Video element
stream -> <video>
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output
Plus other non-MediaStream pipelines:
????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output
Lets take a closer look at the
Video/Canvas MediaStream pipeline
https://buildAR.com
https://buildAR.com
The Video/Canvas MediaStream pipeline
Access the camera (or some other stream) source
a. getUserMedia()
Setup <video> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“video”) (no need to appendChild())
Pipe camera stream into <video>
a. video.src = stream
Setup <canvas> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“canvas”) then appendChild()
Get 2D drawing context
a. canvas.getContext('2d');
Draw <video> frame onto <canvas>
a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixels
a. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cycles
a. fore each(var obj in view) { … }
NOTE: Integrate other streams & sensor data here
Render results
a. using HTML/JS/CSS
b. using another <canvas> and drawImage()
c. using WebGL
d. a combination of all
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
5. Populate a scene with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
Follow along on github: github.com/buildar/getting_started_with_webrtc
https://buildAR.com
The Video/Canvas MediaStream pipeline
Follow along on github: github.com/buildar/getting_started_with_webrtc
Access the camera (or some other stream) source
a. getUserMedia()
Setup <video> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“video”) (no need to appendChild())
Pipe camera stream into <video>
a. video.src = stream
Setup <canvas> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“canvas”) then appendChild()
Get 2D drawing context
a. canvas.getContext('2d');
Draw <video> frame onto <canvas>
a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixels
a. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cycles
a. fore each(var obj in view) { … }
NOTE: Integrate other streams & sensor data here
Render results
a. using HTML/JS/CSS
b. using another <canvas> and drawImage()
c. using WebGL
d. a combination of all
setInterval()
RequestAnimationFrame()
setTimeout()
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.com
Join the W3C's
Augmented Web Community Group
http://w3.org/community/ar

More Related Content

More from Rob Manson

The Power of AR & VR in Learning
The Power of AR & VR in LearningThe Power of AR & VR in Learning
The Power of AR & VR in LearningRob Manson
 
Immersive Sydney - #WebXRWeek
Immersive Sydney - #WebXRWeekImmersive Sydney - #WebXRWeek
Immersive Sydney - #WebXRWeekRob Manson
 
The Extended Reality Landscape #Reality17
The Extended Reality Landscape #Reality17The Extended Reality Landscape #Reality17
The Extended Reality Landscape #Reality17Rob Manson
 
ISMAR17 - Augmenting, Mixing & Extending Reality on the web
ISMAR17 - Augmenting, Mixing & Extending Reality on the webISMAR17 - Augmenting, Mixing & Extending Reality on the web
ISMAR17 - Augmenting, Mixing & Extending Reality on the webRob Manson
 
Immersive Computing @ #YOWConnected 2017
Immersive Computing @ #YOWConnected 2017Immersive Computing @ #YOWConnected 2017
Immersive Computing @ #YOWConnected 2017Rob Manson
 
Immersive - Sydney July 2017
Immersive - Sydney July 2017Immersive - Sydney July 2017
Immersive - Sydney July 2017Rob Manson
 
Computer Vision - now working
 in over 2 Billion Web Browsers!
Computer Vision - now working
 in over 2 Billion Web Browsers!Computer Vision - now working
 in over 2 Billion Web Browsers!
Computer Vision - now working
 in over 2 Billion Web Browsers!Rob Manson
 
The future is un-Real
The future is un-RealThe future is un-Real
The future is un-RealRob Manson
 
Introduction to AR
Introduction to ARIntroduction to AR
Introduction to ARRob Manson
 
Web Standards adoption in the AR market
Web Standards adoption in the AR marketWeb Standards adoption in the AR market
Web Standards adoption in the AR marketRob Manson
 
Now here is everywhere
Now here is everywhereNow here is everywhere
Now here is everywhereRob Manson
 
AR is the UI for Pervasive Computing
AR is the UI for Pervasive ComputingAR is the UI for Pervasive Computing
AR is the UI for Pervasive ComputingRob Manson
 
Your browser can see and hear and ...
Your browser can see and hear and ...Your browser can see and hear and ...
Your browser can see and hear and ...Rob Manson
 
Web Standards based Augmented Reality
Web Standards based Augmented RealityWeb Standards based Augmented Reality
Web Standards based Augmented RealityRob Manson
 
Is that sense-able?
Is that sense-able?Is that sense-able?
Is that sense-able?Rob Manson
 
streetARt case study for ARE2011
streetARt case study for ARE2011streetARt case study for ARE2011
streetARt case study for ARE2011Rob Manson
 
ARSyd March 2011
ARSyd March 2011ARSyd March 2011
ARSyd March 2011Rob Manson
 
Patterns of Interest - ISWAR Seoul 2010
Patterns of Interest - ISWAR Seoul 2010Patterns of Interest - ISWAR Seoul 2010
Patterns of Interest - ISWAR Seoul 2010Rob Manson
 
ARSyd September 2010
ARSyd September 2010ARSyd September 2010
ARSyd September 2010Rob Manson
 

More from Rob Manson (20)

The Power of AR & VR in Learning
The Power of AR & VR in LearningThe Power of AR & VR in Learning
The Power of AR & VR in Learning
 
Immersive Sydney - #WebXRWeek
Immersive Sydney - #WebXRWeekImmersive Sydney - #WebXRWeek
Immersive Sydney - #WebXRWeek
 
The Extended Reality Landscape #Reality17
The Extended Reality Landscape #Reality17The Extended Reality Landscape #Reality17
The Extended Reality Landscape #Reality17
 
ISMAR17 - Augmenting, Mixing & Extending Reality on the web
ISMAR17 - Augmenting, Mixing & Extending Reality on the webISMAR17 - Augmenting, Mixing & Extending Reality on the web
ISMAR17 - Augmenting, Mixing & Extending Reality on the web
 
Immersive Computing @ #YOWConnected 2017
Immersive Computing @ #YOWConnected 2017Immersive Computing @ #YOWConnected 2017
Immersive Computing @ #YOWConnected 2017
 
Immersive - Sydney July 2017
Immersive - Sydney July 2017Immersive - Sydney July 2017
Immersive - Sydney July 2017
 
Computer Vision - now working
 in over 2 Billion Web Browsers!
Computer Vision - now working
 in over 2 Billion Web Browsers!Computer Vision - now working
 in over 2 Billion Web Browsers!
Computer Vision - now working
 in over 2 Billion Web Browsers!
 
The future is un-Real
The future is un-RealThe future is un-Real
The future is un-Real
 
Introduction to AR
Introduction to ARIntroduction to AR
Introduction to AR
 
Web Standards adoption in the AR market
Web Standards adoption in the AR marketWeb Standards adoption in the AR market
Web Standards adoption in the AR market
 
Now here is everywhere
Now here is everywhereNow here is everywhere
Now here is everywhere
 
AR is the UI for Pervasive Computing
AR is the UI for Pervasive ComputingAR is the UI for Pervasive Computing
AR is the UI for Pervasive Computing
 
Your browser can see and hear and ...
Your browser can see and hear and ...Your browser can see and hear and ...
Your browser can see and hear and ...
 
Web Standards based Augmented Reality
Web Standards based Augmented RealityWeb Standards based Augmented Reality
Web Standards based Augmented Reality
 
Is that sense-able?
Is that sense-able?Is that sense-able?
Is that sense-able?
 
What is web3?
What is web3?What is web3?
What is web3?
 
streetARt case study for ARE2011
streetARt case study for ARE2011streetARt case study for ARE2011
streetARt case study for ARE2011
 
ARSyd March 2011
ARSyd March 2011ARSyd March 2011
ARSyd March 2011
 
Patterns of Interest - ISWAR Seoul 2010
Patterns of Interest - ISWAR Seoul 2010Patterns of Interest - ISWAR Seoul 2010
Patterns of Interest - ISWAR Seoul 2010
 
ARSyd September 2010
ARSyd September 2010ARSyd September 2010
ARSyd September 2010
 

Recently uploaded

TheOrmaxSportsAudienceReport2024-MediaHighlights.pdf
TheOrmaxSportsAudienceReport2024-MediaHighlights.pdfTheOrmaxSportsAudienceReport2024-MediaHighlights.pdf
TheOrmaxSportsAudienceReport2024-MediaHighlights.pdfSocial Samosa
 
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...mxmindia
 
Kobe Bryant, The story of the "Black Mamba"
Kobe Bryant, The story of the "Black Mamba"Kobe Bryant, The story of the "Black Mamba"
Kobe Bryant, The story of the "Black Mamba"bmfcavalchini
 
Nicholas Kodomichalos: A Tale of Sporting Brilliance with Nick Kodos
Nicholas Kodomichalos: A Tale of Sporting Brilliance with Nick KodosNicholas Kodomichalos: A Tale of Sporting Brilliance with Nick Kodos
Nicholas Kodomichalos: A Tale of Sporting Brilliance with Nick KodosNicholasKodomichalos1
 
Instruction Manual | Nightforce CFS 6-36x50 F1 | Optics Trade
Instruction Manual | Nightforce CFS 6-36x50 F1 | Optics TradeInstruction Manual | Nightforce CFS 6-36x50 F1 | Optics Trade
Instruction Manual | Nightforce CFS 6-36x50 F1 | Optics TradeOptics-Trade
 
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...mxmindia
 
Euro 2024 Serbia's Return to the European Championship.docx
Euro 2024 Serbia's Return to the European Championship.docxEuro 2024 Serbia's Return to the European Championship.docx
Euro 2024 Serbia's Return to the European Championship.docxEticketing.co
 
IPL 2024 Fixture List Match Dates, Participating Teams, Stadiums, Venues, and...
IPL 2024 Fixture List Match Dates, Participating Teams, Stadiums, Venues, and...IPL 2024 Fixture List Match Dates, Participating Teams, Stadiums, Venues, and...
IPL 2024 Fixture List Match Dates, Participating Teams, Stadiums, Venues, and...Sportsnewstec24
 
NCAA Tournament Free Pick, March 22 From Top March Madness Sports Handicapper
NCAA Tournament Free Pick, March 22 From  Top March Madness Sports HandicapperNCAA Tournament Free Pick, March 22 From  Top March Madness Sports Handicapper
NCAA Tournament Free Pick, March 22 From Top March Madness Sports HandicapperJoe Duffy
 

Recently uploaded (9)

TheOrmaxSportsAudienceReport2024-MediaHighlights.pdf
TheOrmaxSportsAudienceReport2024-MediaHighlights.pdfTheOrmaxSportsAudienceReport2024-MediaHighlights.pdf
TheOrmaxSportsAudienceReport2024-MediaHighlights.pdf
 
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
 
Kobe Bryant, The story of the "Black Mamba"
Kobe Bryant, The story of the "Black Mamba"Kobe Bryant, The story of the "Black Mamba"
Kobe Bryant, The story of the "Black Mamba"
 
Nicholas Kodomichalos: A Tale of Sporting Brilliance with Nick Kodos
Nicholas Kodomichalos: A Tale of Sporting Brilliance with Nick KodosNicholas Kodomichalos: A Tale of Sporting Brilliance with Nick Kodos
Nicholas Kodomichalos: A Tale of Sporting Brilliance with Nick Kodos
 
Instruction Manual | Nightforce CFS 6-36x50 F1 | Optics Trade
Instruction Manual | Nightforce CFS 6-36x50 F1 | Optics TradeInstruction Manual | Nightforce CFS 6-36x50 F1 | Optics Trade
Instruction Manual | Nightforce CFS 6-36x50 F1 | Optics Trade
 
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
Changing mindsets, rebuilding lives: Celebrating 25 years of First World Cup ...
 
Euro 2024 Serbia's Return to the European Championship.docx
Euro 2024 Serbia's Return to the European Championship.docxEuro 2024 Serbia's Return to the European Championship.docx
Euro 2024 Serbia's Return to the European Championship.docx
 
IPL 2024 Fixture List Match Dates, Participating Teams, Stadiums, Venues, and...
IPL 2024 Fixture List Match Dates, Participating Teams, Stadiums, Venues, and...IPL 2024 Fixture List Match Dates, Participating Teams, Stadiums, Venues, and...
IPL 2024 Fixture List Match Dates, Participating Teams, Stadiums, Venues, and...
 
NCAA Tournament Free Pick, March 22 From Top March Madness Sports Handicapper
NCAA Tournament Free Pick, March 22 From  Top March Madness Sports HandicapperNCAA Tournament Free Pick, March 22 From  Top March Madness Sports Handicapper
NCAA Tournament Free Pick, March 22 From Top March Madness Sports Handicapper
 

MediaStream processing pipelines on the Augmented Web Platform

  • 1. https://buildAR.com - image credit It's alive! MediaStream processing pipelines on the Augmented Web Platform
  • 2. Who's making these outlandish claims? Rob Manson @nambor CEO of MOB-labs the creators of buildAR.com Chair of the W3C's Augmented Web Community Group Invited Expert with the ISO, W3C & the Khronos Group Author of “Getting started with WebRTC” - (Packt Publishing) https://buildAR.com
  • 3. What is the Augmented Web? https://buildAR.com
  • 4. It's what comes after HTML5 https://buildAR.com
  • 5. The web is evolving... 1992-2000 WWW: Page based websites key innovation: HREF 2000-2013 Web 2.0: AJAX driven social services key innovation: XHR 2013-... AWE: Sensor driven real-time web of things key innovation: ArrayBuffer http://mob-labs.com : image credit
  • 9. Test your device at: theAWEsomeWEB.com https://buildAR.com
  • 10. So what the hell is a Stream processing pipeline? https://buildAR.com
  • 11. https://buildAR.com Stream processing pipelines 1. Get Stream 2. Connect to a Stream pipeline A way to connect a Stream to an ArrayBuffer 3. Get ArrayBuffer 5. Populate a scene Array with Array Buffer Views 6. Move through Views to process data 7. Output events and metadata 8. Update UI and/or send requests
  • 12. Then what is a MediaStream processing pipeline? https://buildAR.com
  • 13. https://buildAR.com MediaStream processing pipelines 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here!
  • 14. https://buildAR.com This is where the APIs are still evolving 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  • 15. https://buildAR.com This is where the Augmented Web comes to life 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  • 16. https://buildAR.com This is where the fun is! 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  • 18. https://buildAR.com Browser's have always had a “binary barrier” The Video element stream -> <video>
  • 19. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
  • 20. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output
  • 21. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output
  • 22. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output Plus other non-MediaStream pipelines: ????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output
  • 23. Lets take a closer look at the Video/Canvas MediaStream pipeline https://buildAR.com
  • 24. https://buildAR.com The Video/Canvas MediaStream pipeline Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! Follow along on github: github.com/buildar/getting_started_with_webrtc
  • 25. https://buildAR.com The Video/Canvas MediaStream pipeline Follow along on github: github.com/buildar/getting_started_with_webrtc Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all setInterval() RequestAnimationFrame() setTimeout()
  • 26. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  • 27. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  • 28. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  • 29. https://buildAR.com Join the W3C's Augmented Web Community Group http://w3.org/community/ar

Editor's Notes

  1. http://www.khronos.org/streaminput/