My YGLF 2015 Talk Recently browsers have been introduced with interesting tools for work with binary stream of data. Technologies like XHR2 and File API allow us to fetch binary blobs from urls and the file system, and from there, a whole new world is opened before us: we can use Media Source Extensions to implement live streaming audio/video protocols like DASH and HLS, WebRTC to transmit data P2P bittorrent style, WebGL to draw shapes from arrays of binary position data, and more. Furthermore, using technologies like asm.js and tools like Emscripten to transpile C++ code to JavaScript, we can do amazing things with all of the above, and achieve superior performance. In this talk we'll explore some of these techniques, and learn about how we are solving interesting problems with them.

1. BINARY DATA ADVENTURES
IN BROWSER JAVASCRIPT
OR HILTCH / CTO @ STREAMRAIL

2. INTRO
• Background
• Evolution in the browser
• The Binary Arsenal in JS (type system,
inputs, outputs)
• Solving Cool, Practical Problems
00110110011100100101010100100
10101011101000001010100101010
01010010111010101001010101010
010101011111001010100010101110
0101010111101000000101010100
01011001010101010111100001010

3. TRADITIONALLY
• Outside of JS:
Low level stuff: Compression (gzip/deflate), Bitfields (ACLs),
Networks (CRC32), Graphics (algorithms, positioning), ...
• Inside of JS:
Solving weird issues

4. $http.post('/someUrl', {score:42}).
success(function(data, status, headers, config) {
// not today :-(
}).
error(function(data, status, headers, config) {
// data=Object {‘error’: ‘Float64 is not int’}
}

5. $http.post('/someUrl', {score:42}).
success(function(data, status, headers, config) {
// not today :-(
}).
error(function(data, status, headers, config) {
// data=Object {‘error’: ‘Float64 is not int’}
}

6. typeof 3.14159 === typeof 3

7. In JS, all numbers are of type Float64
But all bitwise ops are Int32!
Bitwise OR: number | 0
Bitwise NOT: ~~number
Shift: number >> 0
parseInt: parseInt(number, 10)
floor: Math.floor(number)
modulo: number - number % 1

9. BROWSERS, EVOLVED
• Type system: Blob, ArrayBuffer, TypedArray, DataView
• Inputs:
Input methods: XHR 2, File API, Canvas, WebSockets, WebRTC, ...
• Outputs:
MSE, Canvas, WebSockets, WebRTC, WebGL, ...

10. WHAT MAKES
TYPED ARRAYS FAST
• Passed to native interfaces completely AS IS (direct memory access)
• Native “endianness” - watch out!
• DataView adapter
Byte order
0x12345678 Hex inside a Uint32Array of 4
Little-endian: 78 56 34 12
12 34 56 78Big-endian:

11. STREAMING VIDEO
WITH JS
• No plugins (Netflix: silverlight, Facebook: flash)
• MSE: Pure JS DASH/HLS (YouTube)
• DRM Support (EME)
• MSE + WebRTC: Bittorrent, in the browser! (WebTorrent)

12. ‘USE ASM’
sys/libkern/strlen.c
size_t strlen(const char *str) {
const char *s;
for (s = str; *s; ++s);
return(s - str);
}
asm.js
function strlen(ptr) {
ptr = ptr|0;
var curr = 0;
curr = ptr;
while (MEM8[curr]|0 != 0) {
curr = (curr + 1)|0;
}
return (curr - ptr)|0;
}

13. a subset of JS for handling numbers faster
• OdinMonkey - AOT in FF (fallback to IonMonkey JIT)
• Support by all major browsers (to an extent)
• Typed Array used as “memory”
• All add/sub are 32 bit (number | 0)
• DI: function(stdlib, foreign, heap)
‘USE ASM’

16. KRIPKEN TO THE RESCUE
C/C++ > LLVM > Emscripten > asm.js!
• Huge code bases (SQLite, BananaBread,J2ME VM, …)
- close to impossible by hand
• Enjoy Clang/LLVM optimizations
- decades of work done to optimize compiled code
• OpenGL > WebGL “for free”
- directly map a lot of OpenGL ES 2.0 command to WebGL
- not only for graphics rendering, also for GPU offloading

17. • Use Docker to run Emscripten
- lots of software (emscripten, emscripten-fastcomp, emscripten-
fastcomp-clang, llvm clang)
- build your own or use one from DockerHub
KRIPKEN TO THE RESCUE

18. emscripten in real life

19. • Obtain bytes of video (and audio) by downloading them
• Decode the video (and audio) using native decoders in the browser/OS
• Animate the frames to create the video experience
VIDEO TAG EMULATION

20. • Obtain bytes of video (and audio) by downloading them
> XHR 2.0, response type “arraybuffer”
• Decode the video (and audio) using native decoders in the browser/OS
> Cross compile a native decoder to JS using Emscripten
• Animate the frames to create the video experience
> For each decoded pixel, create a 2D Texture, transform it to RGBA using
a fragment shader, and render it with WebGL using canvas
(“experimental-webgl”).
VIDEO TAG EMULATION

21. YUV -> RGB
• Luma + (2 * Chroma) vs
3 * (Chroma + Luma)
• Different needs, different inputs
• Color space conversion heavily
computational (floating point coefficients
matrix multiplication)

22. WEBGL RASTERIZATION

23. function yuv2canvas(buffer, width, height) {
var lumaSize = width * height;
var chromaSize = lumaSize >> 2;
webGLCanvas.YTexture
.fill(buffer.subarray(0, lumaSize));
webGLCanvas.UTexture
.fill(buffer.subarray(lumaSize, lumaSize + chromaSize));
webGLCanvas.VTexture
.fill(buffer.subarray(lumaSize + chromaSize,
lumaSize + 2 * chromaSize));
webGLCanvas.drawScene();
}
4:2:0
=
+

24. precision highp float;
varying highp vec2 vTextureCoord;
uniform sampler2D YTexture;
uniform sampler2D UTexture;
uniform sampler2D VTexture;
const mat4 YUV2RGB = mat4 (
1.1643828125, 0 , 1.59602734375, -.87078515625,
1.1643828125, -0.39176171875, -0.81296875 , .52959375,
1.1643828125, 2.017234375 , 0 , -1.081390625,
0 , 0 , 0 , 1
);
void main(void) {
gl_FragColor = vec4(texture2D(YTexture, vTextureCoord).x,
texture2D(UTexture, vTextureCoord).x,
texture2D(VTexture, vTextureCoord).x,
1)
* YUV2RGB;
}
<SCRIPT TYPE="X-SHADER/X-FRAGMENT">

25. LIVE DEMO

26. THANK YOU!
OR HILTCH / CTO @ STREAMRAIL