Talk di Alberto Mancini alla DevFest 2012 @ Firenze

1. Unconventional webapps with
GWT/Elemental, WebRTC &
WebGL
Alberto Mancini
alberto@jooink.com
Blog: http://jooink.blogspot.com
Online Demo:
http://www.jooink.com/experiments/DevFest2012

2. GWT 2.5 Elemental
Elemental is a new library for fast, lightweight,
and "to the metal" web programming in GWT.
Elemental includes every HTML5 feature, ... DOM access
... WebGL, WebAudio, WebSockets, WebRTC, Web
Intents, Shadow DOM, the File API, and more.
... we generate Java code directly from the WebIDL files
used by JavaScript engines. (like DART)

3. Unconventional
... insomma qualcosa che non ci sembrava
possibile fare con il solo browser, tipo accedere
a devices attaccati al 'client', una webcam
per esempio.
L'idea e' farsi una foto ...

4. WebRTC
WebRTC (Web Real-Time Communication) is an API
definition being drafted by W3C and IETF.
The goal of WebRTC is to enable applications such as
voice calling, video chat and P2P file sharing without
plugins.
Specification & Up to date info: http://www.webrtc.org
Support: Chrome 22+, ... well work-in-progress at least for firefox.
what we need: getUserMedia (chrome !!)

5. WebRTC (with Elemental)
Video in a canvas
final VideoElement videoElement =
Browser.getDocument().createVideoElement();
final CanvasElement canvas =
Browser.getDocument().createCanvasElement();
final CanvasRenderingContext2D ctx2d =
(CanvasRenderingContext2D)canvas.getContext("2d");
...
//repeatedly, e.g. in a Timer or RepeatingCommand
ctx2d.drawImage(videoElement, 0, 0);
Take snapshot as easy as:
Image img = new Image(canvas.toDataURL("png"));

6. WebRTC (with Elemental)
binding <video/> to userMedia (search for facelogin on googlecode)
public void
bindVideoToUserMedia(final VideoElement video, final EventListener l) {
final Mappable map = (Mappable) JsMappable.createObject();
map.setAt("video", true);
Browser.getWindow().getNavigator().webkitGetUserMedia(map,
new NavigatorUserMediaSuccessCallback() {
public boolean onNavigatorUserMediaSuccessCallback
(LocalMediaStream stream) {
setVideoSrc(video, stream);
video.play(); ...
}, new NavigatorUserMediaErrorCallback() {...});
}
private static native void setVideoSrc( VideoElement v, LocalMediaStream s) /*-{
v.src = window.webkitURL.createObjectURL(s);
}-*/;

7. WebGL
WebGL (Web Graphics Library) is a JavaScript API for
rendering interactive 3D graphics and 2D graphics within
any compatible web browser without the use of plug-ins.
Specification: http://www.khronos.org/webgl/
Support: http://caniuse.com/webgl
Chrome 22+, FireFox 15+, Safari 6+, Opera 12+ (partial)

8. WebGL (with Elemental)
Video in a canvas (3d rendering context)
...
WebGLRenderingContext ctx3d =
(WebGLRenderingContext)canvas.getContext("experimental-webgl");
Drawing is a bit harder:
create a texture, create a background rectangle,
use the video as a texture
ctx3d.texImage2D(WebGLRenderingContext.TEXTURE_2D, 0,
WebGLRenderingContext.RGBA, WebGLRenderingContext.RGBA,
WebGLRenderingContext.UNSIGNED_BYTE, videoElement);

9. WebGL (with Elemental)
shaders
Vertex Shader Fragment Shader
precision mediump float; precision mediump float;
attribute vec2 a_position; uniform sampler2D u_image;
attribute vec2 a_texCoord; varying vec2 v_texCoord;
uniform vec2 u_resolution;
varying vec2 v_texCoord; void main() {
void main() { gl_FragColor =
vec2 zeroToOne = a_position / u_resolution; texture2D(u_image, v_texCoord);
vec2 zeroToTwo = zeroToOne * 2.0; }
vec2 clipSpace = zeroToTwo - 1.0;
gl_Position = vec4(clipSpace * vec2(1, -1), 1, 1);
v_texCoord = a_texCoord;
}
from html5rocks.com's WebGL Fundamentals ... probably :o

10. WebGL (with Elemental)
shaders
Fragment Shader Fragment Shader
...
// Sepia tone precision mediump float;
// Convert to greyscale using NTSC weightings uniform sampler2D u_image;
float grey = dot(texture2D(u_image, uniform sampler2D SECOND_u_image;
v_texCoord).rgb, vec3(0.299, 0.587, 0.114)); uniform float alpha;
gl_FragColor = varying vec2 v_texCoord;
vec4(grey * vec3(1.2, 1.0, 0.8), 1.0); void main(void) {
... vec4 prev = texture2D(SECOND_u_image,
// Negative vec2(v_texCoord.s,1.0-v_texCoord.t));
vec4 texColour = vec4 cur = texture2D(u_image, v_texCoord);
texture2D(u_image, v_texCoord); gl_FragColor = alpha*prev + (1.0-alpha)*cur;
gl_FragColor = vec4(1.0 - texColour.rgb, 1.0); }
from http://r3dux.org/2011/06/glsl-image-processing/

11. WebGL (with Elemental)
3D
WebGL e' una libreria 2D !!
Trasformazioni prospettiche (proiezioni), frustum, modelview
vanno implementate.
VertexShader: applichera' le trasformazioni (matrici) che noi gli
forniremo, vertice per vertice
FragmentShader: usera', pixel-per-pixel, le informazioni
calcolate vertice-per-vertice (ed interpolate dal 'sistema')
per assegnare il colore

12. WebGL (with Elemental)
3D
double a = (right + left) / (right - left);
double b = (top + bottom) / (top - bottom);
double c = (farPlane + nearPlane) / (farPlane - nearPlane);
double d = (2 * farPlane * nearPlane) / (farPlane - nearPlane);
double x = (2 * nearPlane) / (right - left);
double y = (2 * nearPlane) / (top - bottom);
double[] perspectiveMatrix = new double[]{
x, 0, a, 0,
0, y, b, 0,
0, 0, c, d,
0, 0, -1, 0
};
....
ctx3d.uniformMatrix4fv(pMatrixUniform, false, Utilities.createArrayOfFloat32(perspectiveMatrix));
...
private native static Float32Array createFloat32Array(JsArrayOfNumber a) /*-{
return new Float32Array(a);
}-*/;

13. WebGL (with Elemental)
3D/Wavefront-OBJ
Loading Wavefront-obj files ...
parser in java facile da trovare in rete
ClientBundle DataResource perfetto per il loading
asincrono dei modelli
Javascript per calcolare le normali se non sono nel
modello

14. Unconventional
Tutto quello che abbiamo visto fino ad ora si poteva fare,
probabilmente con lo stesso sforzo, direttamente in javascript.
Ma usare GWT ci da o no una marcia in piu' ?
Nota:
non stiamo dicendo che quello che segue non si potesse fare in
javascript, ndr: Turing completeness, se ce ne fosse bisogno !

15. GWT
Compiler: Java -> Javascript
Javascript as a target language

16. NyARToolkit
ARToolKit is a computer tracking library for creation of strong augmented
reality applications that overlay virtual imagery on the real world.
NyARToolKit is an ARToolkit class library released for virtual machines,
particularly those that host Java, C# and Android.
Up to date info
http://nyatla.jp/nyartoolkit/wp/?page_id=198
Support:
Java ... "Write once, run anywhere" (WORA), or sometimes write once, run
everywhere (WORE) ... in my browser ?!?!?!

17. NyARToolkit con GWT
http://jooink.blogpsot.com
super-source tag
per InputStream & altre parti della JRE non disponibili in GWT
byte[] raster;
UInt8ClampedArray image = ImageData.getData();
private static native byte[] toArrayOfBytes(Uint8ClampedArray a) /*-{
return a;
}-*/;
Documented in JAPANESE !!!

18. How It Works
cam WebRTC <video/>
canvas ImageData
Elemental/jsni
WebGL + model
video
GWT(NyARToolkit) byte[]
mv matrix

19. Demo http://www.jooink.com/experiments/DevFest2012