WebRTC
... GWT & in-browser computation
Alberto Mancini - alberto@jooink.com
Francesca Tosi - francesca@jooink.com
WebRTC
Plug-in free real-time communication …
WebRTC is a free, open project that enables web browsers
with Real-Time Comm...
WebRTC
Just another JS API
or
WebRTC is a new front in the long
war for an open and
unencumbered web
Brendan Eich (Mozilla...
WebRTC
Just another JS API
or
WebRTC is a new front in the long
war for an open and
unencumbered web
Brendan Eich (Mozilla...
Once upon a time … browsers
(js-)applications
were sandboxed and
kept far away from
the hosting
computer’s
resources.
Once upon a time … browsers
(js-)applications
were sandboxed and
kept far away from
the hosting
computer’s
resources.
Browser Piercing (aka HTML5)
a Javascript Application cannot
➔ access the filesystem
➔ open f/d socket connections
➔ use g...
Browser Piercing (aka HTML5)
a Javascript Application cannot
➔ access the filesystem → and then FileAPI
➔ open f/d socket ...
Browser Piercing
a Javascript Application cannot
➔ acquire audio and video
➔ communicate P2P
Browser Piercing
a Javascript Application cannot
➔ acquire audio and video
➔ communicate P2P
WebRTC … secure enough?
“… WebRTC has encryption baked right into
it; there's actually no way to send
unencrypted media us...
WebRTC … secure enough?
But ...
“If the developers fail to carefully consider the
security implications of their choices, ...
WebRTC … secure enough?
Do not ...
“... it would be very easy to inadvertently click
on something that gave camera or micr...
WebRTC … secure enough?
See also:
WebRTC: APIs, Protocols and Security
Considerations - Part 1 / Part 2
WebRTC Security an...
WebRTC across platforms
● Chrome (mobile too)
● Firefox (mobile too)
● Opera
● Native Java and Obj-C
WebRTC JS-API
Some code
…
WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
WebRTC JS-API
Constraints
● Controls the contents of the MediaStream
● Media type, resolution, frame rate
var constraints ...
WebRTC JS-API
Constraints
● Controls the contents of the MediaStream
● Media type, resolution, frame rate
var constraints ...
WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
WebRTC JS-API
errorCallback
function errorCallback(error) {
console.log("error: ", error);
}
;-)
WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
WebRTC JS-API
successCallback
function successCallback(stream) {
...
}
stream: MediaStream; a flux of audio- or video-rela...
WebRTC JS-API
successCallback
function successCallback(stream) {
var video = ...
video.src =
window.URL.createObjectURL(st...
WebRTC JS-API
full sample
var constraints = {video: true};
function successCallback(stream) {
var video = document.querySe...
WebRTC JS-API
Live Demo - 1
http://goo.gl/7X91Ie
WebRTC JS-API
RTCPeerConnection
→
getUserMedia
+
RTCPeerConnection
←
WebRTC JS-API
RTCDataChannel
Bidirectional communication of arbitrary data
var position = {x: 3.0, y: -2.0};
...
var messa...
RTCPeerConnection sample
pc = new RTCPeerConnection(null);
pc.onaddstream = gotRemoteStream;
pc.addStream(localStream);
pc...
RTCPeerConnection
Live Demo - 2
https://apprtc.appspot.com
RTCPeerConnection
RTCPeerConnection
RTCPeerConnection
STUN
TURN
WebRTC made easy
...
● Video chat:
○ SimpleWebRTC
○ easyRTC
○ webRTC.io
● Peer-to-peer data:
○ PeerJS
○ Sharefest
Our target
Our target
● getUserMedia: gives us real time stream from the camera
● we can draw the <video/> into a <canvas/>
● from th...
Computing in the browser
goals
- Almost native performance
- Manage complex algorithms and applications
on the browser
Our target
cam WebRTC <video/>
canvas ImageData
UInt8ArrayGWT(NyARToolkit)mv matrix
WebGL OBJ Model
Loader3D Shaders
WebGL...
Computing in the browser
constraints
- reuse existing code
- cross-browser (as much as possible)
Computing in the browser
our approach
- javascript as a target language (GWT)
- hand written optimized pieces of code
GWT
GWT Web Toolkit
- java to javascript (optimizing) compiler
- emulated jre
- widget set
- develop in java and execute i...
JooinK’s WebRTC sample
Sample
http://picshare.jooink.com
Demo …
PicShare
Tecnologies
● WebGL for 3D graphics
● WebRTC for MediaSteams
● ImgUR for pictures store
● AppEngine to store indi...
FOSS
gwt-nyartoolkit.googlecode.com
gwt-webgl.googlecode.com
The strategy: compile a java-lib with GWT into javascript
What’s next
Challanges
● real-time on mobile apps
● real world recognition: computer vision
(BoofCV)
What’s next
Strategy
● use TypedArrays everywhere
● offload computation to the graphic
accelerator through WebGL
● help th...
References
● http://io13webrtc.appspot.com/#1
(where we got inspiration and most of the pictures)
● http://www.youtube.com...
References
● http://www.jooink.com
● http://jooink.blogspot.com
● http://www.mokabyte.it
That’s all folks!!!
Francesca Tosi
francesca@jooink.com
Alberto Mancini
alberto@jooink.com
Markers - use at picshare.jooink.com
Upcoming SlideShare
Loading in …5
×

WebRTC ... GWT & in-browser computation

1,005 views

Published on

Alberto Mancini - Francesca Tosi
WebRTC ... GWT & in-browser computation
Talk @ Open Source Day 2013

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,005
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

WebRTC ... GWT & in-browser computation

  1. 1. WebRTC ... GWT & in-browser computation Alberto Mancini - alberto@jooink.com Francesca Tosi - francesca@jooink.com
  2. 2. WebRTC Plug-in free real-time communication … WebRTC is a free, open project that enables web browsers with Real-Time Communications (RTC) capabilities via simple Javascript APIs. source: webrtc.org
  3. 3. WebRTC Just another JS API or WebRTC is a new front in the long war for an open and unencumbered web Brendan Eich (Mozilla CTO and inventor of JavaScript) ?
  4. 4. WebRTC Just another JS API or WebRTC is a new front in the long war for an open and unencumbered web Brendan Eich (Mozilla CTO and inventor of JavaScript) ?
  5. 5. Once upon a time … browsers (js-)applications were sandboxed and kept far away from the hosting computer’s resources.
  6. 6. Once upon a time … browsers (js-)applications were sandboxed and kept far away from the hosting computer’s resources.
  7. 7. Browser Piercing (aka HTML5) a Javascript Application cannot ➔ access the filesystem ➔ open f/d socket connections ➔ use graphics accelerator ➔ span multiple threads
  8. 8. Browser Piercing (aka HTML5) a Javascript Application cannot ➔ access the filesystem → and then FileAPI ➔ open f/d socket connections → and then WebSockets ➔ use graphics accelerator → and then WebGL ➔ span multiple threads → and then WebWorkers
  9. 9. Browser Piercing a Javascript Application cannot ➔ acquire audio and video ➔ communicate P2P
  10. 10. Browser Piercing a Javascript Application cannot ➔ acquire audio and video ➔ communicate P2P
  11. 11. WebRTC … secure enough? “… WebRTC has encryption baked right into it; there's actually no way to send unencrypted media using a WebRTC implementation … both Chrome and Firefox implement.”
  12. 12. WebRTC … secure enough? But ... “If the developers fail to carefully consider the security implications of their choices, the safeguards mandated by the specification will not be enough to guarantee the security of their WebRTC-based applications and the users.”
  13. 13. WebRTC … secure enough? Do not ... “... it would be very easy to inadvertently click on something that gave camera or microphone control to someone I don't know and don't care to know.” Courtney Sato - nerd queen @ConstellationRG
  14. 14. WebRTC … secure enough? See also: WebRTC: APIs, Protocols and Security Considerations - Part 1 / Part 2 WebRTC Security and Confidentiality Security Considerations for WebRTC WebRTC Security Architecture
  15. 15. WebRTC across platforms ● Chrome (mobile too) ● Firefox (mobile too) ● Opera ● Native Java and Obj-C
  16. 16. WebRTC JS-API Some code …
  17. 17. WebRTC JS-API Acquiring video and audio navigator.getUserMedia(constraints, successCallback, errorCallback);
  18. 18. WebRTC JS-API Acquiring video and audio navigator.getUserMedia(constraints, successCallback, errorCallback);
  19. 19. WebRTC JS-API Constraints ● Controls the contents of the MediaStream ● Media type, resolution, frame rate var constraints = {video: true}; video: { mandatory: { ... }, optional [{... }] }
  20. 20. WebRTC JS-API Constraints ● Controls the contents of the MediaStream ● Media type, resolution, frame rate var constraints = {video: true}; video: { mandatory: { chromeMediaSource: ‘screen’ }, optional [{... }] }
  21. 21. WebRTC JS-API Acquiring video and audio navigator.getUserMedia(constraints, successCallback, errorCallback);
  22. 22. WebRTC JS-API errorCallback function errorCallback(error) { console.log("error: ", error); } ;-)
  23. 23. WebRTC JS-API Acquiring video and audio navigator.getUserMedia(constraints, successCallback, errorCallback);
  24. 24. WebRTC JS-API successCallback function successCallback(stream) { ... } stream: MediaStream; a flux of audio- or video-related data.
  25. 25. WebRTC JS-API successCallback function successCallback(stream) { var video = ... video.src = window.URL.createObjectURL(stream); } (W3C File API)
  26. 26. WebRTC JS-API full sample var constraints = {video: true}; function successCallback(stream) { var video = document.querySelector("video"); video.src = window.URL.createObjectURL(stream); } function errorCallback(error) { console.log("navigator.getUserMedia error: ", error); } navigator.getUserMedia(constraints, successCallback, errorCallback);
  27. 27. WebRTC JS-API Live Demo - 1 http://goo.gl/7X91Ie
  28. 28. WebRTC JS-API RTCPeerConnection → getUserMedia + RTCPeerConnection ←
  29. 29. WebRTC JS-API RTCDataChannel Bidirectional communication of arbitrary data var position = {x: 3.0, y: -2.0}; ... var message = {txt: ‘...’, timestamp: ...};
  30. 30. RTCPeerConnection sample pc = new RTCPeerConnection(null); pc.onaddstream = gotRemoteStream; pc.addStream(localStream); pc.createOffer(gotOffer); function gotOffer(desc) { pc.setLocalDescription(desc); sendOffer(desc); } function gotAnswer(desc) { pc.setRemoteDescription(desc); } function gotRemoteStream(e) { attachMediaStream(remoteVideo, e.stream); }
  31. 31. RTCPeerConnection Live Demo - 2 https://apprtc.appspot.com
  32. 32. RTCPeerConnection
  33. 33. RTCPeerConnection
  34. 34. RTCPeerConnection
  35. 35. STUN
  36. 36. TURN
  37. 37. WebRTC made easy ... ● Video chat: ○ SimpleWebRTC ○ easyRTC ○ webRTC.io ● Peer-to-peer data: ○ PeerJS ○ Sharefest
  38. 38. Our target
  39. 39. Our target ● getUserMedia: gives us real time stream from the camera ● we can draw the <video/> into a <canvas/> ● from the canvas we can extract pixel data getImageData() and then elaborate.
  40. 40. Computing in the browser goals - Almost native performance - Manage complex algorithms and applications on the browser
  41. 41. Our target cam WebRTC <video/> canvas ImageData UInt8ArrayGWT(NyARToolkit)mv matrix WebGL OBJ Model Loader3D Shaders WebGL video effects Video
  42. 42. Computing in the browser constraints - reuse existing code - cross-browser (as much as possible)
  43. 43. Computing in the browser our approach - javascript as a target language (GWT) - hand written optimized pieces of code
  44. 44. GWT GWT Web Toolkit - java to javascript (optimizing) compiler - emulated jre - widget set - develop in java and execute in javascript
  45. 45. JooinK’s WebRTC sample Sample http://picshare.jooink.com Demo …
  46. 46. PicShare Tecnologies ● WebGL for 3D graphics ● WebRTC for MediaSteams ● ImgUR for pictures store ● AppEngine to store indices ● GWT as the glue
  47. 47. FOSS gwt-nyartoolkit.googlecode.com gwt-webgl.googlecode.com The strategy: compile a java-lib with GWT into javascript
  48. 48. What’s next Challanges ● real-time on mobile apps ● real world recognition: computer vision (BoofCV)
  49. 49. What’s next Strategy ● use TypedArrays everywhere ● offload computation to the graphic accelerator through WebGL ● help the JIT compiler/optimizer with asm.js
  50. 50. References ● http://io13webrtc.appspot.com/#1 (where we got inspiration and most of the pictures) ● http://www.youtube.com/watch?v=p2HzZkd2A40 ● http://www.webrtc.org (spec)
  51. 51. References ● http://www.jooink.com ● http://jooink.blogspot.com ● http://www.mokabyte.it
  52. 52. That’s all folks!!! Francesca Tosi francesca@jooink.com Alberto Mancini alberto@jooink.com
  53. 53. Markers - use at picshare.jooink.com

×