This document discusses working with massive data in JavaScript, including Blobs, File APIs, WebSockets, Workers, and real-time communication. It covers how to: - Create and use Blobs to store file-like objects in JavaScript - Read and parse large files by slicing them into parts for parallel processing - Send binary data like files over WebSockets using Blobs and ArrayBuffers - Offload CPU-intensive tasks to Workers to avoid blocking the browser - Establish real-time data channels between peers using WebRTC The document provides code examples for working with Blobs, FileReader, WebSockets, Workers, getUserMedia, AudioContext, and real

1. Massive Data
in JavaScript
by Sergiu Gordienco, 5 July 2015

2. What is Blob ?
How it is used
in JavaScript
var aFileParts = ['<a id="a"><b id="b">hey!</b></a>'];
// the blob
var oMyBlob = new Blob(aFileParts, {type : 'text/html'});

3. Easily work with streams
with buffers & don’t care
about joining data while
we receive it
var aFileParts = ['<a id="a"><b id="b">hey!</b></a>'];
// the blob
var oMyBlob = new Blob(aFileParts, {type : 'text/html'});

4. Blobs can be converted
into data resources with
available URL
var blobURL = window.URL.createObjectURL(new Blob(...))
// and can be used as simple resources
var req = new XMLHttpRequest(); req.onload = calback_f;
req.open('get', blobURL); req.send( .. or send blob ..);

5. Lets start from
popular APIs

6. JavaScript
File API
We get files from <input type=”file” />, and
Drag & Drop events

7. We can get File as
» String / Array .. etc
» Blob
» Stream
var reader = new FileReader();
reader.addEventListener("loadend", callback_f );
reader.readAsArrayBuffer(file_object);

8. You want to parse
2Gb of data?
slice it, parse by parts
var reader = new FileReader(); reader.addEventListener("loadend", callback_f );
reader.readAsArrayBuffer(
file_object.slice(0, 4096) // reading first 4Kb
);

9. JavaScript
WebSocket API
Events: onopen, onerror, onclose, onmessage
Methods: send(), close()
Important Attrs: url, readyState, bufferedAmount

10. void send(
in DOMString data
);
void send(
in ArrayBuffer data
);
void send(
in Blob data
);
When you have possibility use blobs for sending
files, and arraybuffer for stream data,
do not forget about bufferedAmount

11. Huge Algorithm?
browser blocked?
performance issues?
:’( My code works,
just wait to compile 30Mb of CSV file

12. Workers
or
Waiters
CPU is available and we will work :P

13. JavaScript
Workers
parallel processes
… factory495 = new Worker(..), factory496 =
new Worker(..), factory497 = new Worker( ...

14. var worker = new Worker("ak47-vs-mk4.js");
var worker = new Worker("my-csv-builder.js");
}:[ Hey !!! how about this??
My algorithm is in the memory!!!
Starting a parallel worker

15. var worker = new Worker("ak47-vs-mk4.js");
var worker = new Worker("my-csv-builder.js");
}:[ Hey !!! how about this??
My algorithm is in the memory!!!
Starting a parallel worker

16. var blob = new Blob(["... your javascript code ..."
], {type: 'text/javascript'});
// webkit handling
var URL = window.URL || window.webkitURL;
// create the virtual file
var code = URL.createObjectURL(blob);
// create the worker
var worker = new Worker(code);
// for communication use code below, direct contact is not allowed
self.addEventListener('message', messageReceived);
function messageReceived(e) {
self.postMessage('I am HERE !');
}

17. And remeber
every Worker can
have subWorkers
… subFactory495 = new Worker(..),
subFactory496 = new Worker(..), subFac...

18. RTC Data Channelvar peerConnection = new RTCPeerConnection();
// Establish your peer connection using your signaling channel here
var dataChannel =
peerConnection.createDataChannel("myLabel", dataChannelOptions);
dataChannel.onerror = function (error) {
console.log("Data Channel Error:", error);
};
dataChannel.onmessage = function (event) {
console.log("Got Data Channel Message:", event.data);
};
dataChannel.onopen = function () {
dataChannel.send("Hello World!");
};
dataChannel.onclose = function () {
console.log("The Data Channel is Closed");
};

19. Nota Bene:
Firefox supports chunks with maximum 16Kb size
SCTP ( Stream Control Transmission Protocol )
ordered - guarantee order or not
maxRetransmitTime - maximum time to try and retransmit a failed
message (forces unreliable mode)
maxRetransmits - maximum number of times to try and
retransmit a failed message (forces unreliable mode)
protocol - subprotocol to be used
id - provide your own ID for the channel

20. navigator.getUserMedia = navigator.getUserMedia ||
navigator.webkit... || navigator.moz... || navigator.ms...;
audioContext = window.AudioContext || window.webkit… || ...
GetUserMedia
AudioContext
recording audio data

21. callback_success retrieve stream object
callback_error retrieve error object
Requesting Audio Stream
navigator.getUserMedia(
{audio:true},
callback_success,
callback_error
);

22. after we set recorder.onaudioprocess
we will connect recorder using
volume.connect (recorder);
recorder.connect (context.destination);
After retrieving Audio Stream
// build Audio context
// create an audioInput as a stream source
// create the gain mode ( volume / createGain )
// connect audioInput to gain mode
// build an recorder using createJavaScriptNode
bufferSize, numInputChannels, numOutputChannels

25. Allowed BufferSize
256, 512, 1024, 2048, 4096, 8192, or 16384

26. From the start we will describe some
small functions that we will use often
Now we are ready to
record sound
and save it in a file

32. Thanks :)