JavaScript Modules Done Right Client & server-side code organization

1. JavaScript Modules
Done right
Client & server-side code organization
meetjs · November 2011 · Warsaw, Poland

2. Mariusz Nowak
mariusz@medikoo.com
@medikoo
github.com/medikoo
JavaScript Programmer at Roche Ltd.

3. Browser
In the beginning there was Browser.
How we managed large code bases (?)

4. Browser
In the beginning there was Browser.
How we managed large code bases (?)
• Files concatenation

5. Browser
Written usually this style:
MAIN.module = (function() {
var privateVar = '..';
var privateMethod = function (args) {
// ...
};
return {
publicProperty: '..',
publicMethod: function () {
// ...
}
};
}());

6. Browser
In the beginning there was Browser.
How we managed large code bases (?)
• Files concatenation

7. Browser
In the beginning there was Browser.
How we managed large code bases (?)
• Files concatenation
• Module pattern

8. Server
Beginning of 2009
• Growing number of interpreters: Rhino, Spidermonkey, V8, JSCore
• No cross-interpreter standards
• There are programmers who want to write server-side JavaScript but
want to have their code runnable on any engine

9. Server
Beginning of 2009
• Growing number of interpreters: Rhino, Spidermonkey, V8, JSCore
• No cross-interpreter standards
• There are programmers who want to write server-side JavaScript but
want to have their code runnable on any engine
• There’s a need for standard library API that would allow JavaScript
ecosystem grow (as it happened for Ruby, Python or Java)

10. Server
Beginning of 2009
• Growing number of interpreters: Rhino, Spidermonkey, V8, JSCore
• No cross-interpreter standards
• There are programmers who want to write server-side JavaScript but
want to have their code runnable on any engine
• There’s a need for standard library API that would allow JavaScript
ecosystem grow (as it happened for Ruby, Python or Java)
• ServerJS initiative is announced, renamed later into CommonJS
-> http://www.blueskyonmars.com/2009/01/29/what-server-side-javascript-needs/

11. Server
CommonJS presents Modules specification ->
http://www.commonjs.org/specs/modules/1.0/

12. Server
CommonJS presents Modules specification ->
http://www.commonjs.org/specs/modules/1.0/
add.js
exports.add = function() {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
increment.js
var add = require('add').add;
exports.increment = function(val) {
return add(val, 1);
};
program.js
var inc = require('increment').increment;
var a = 1;
inc(a); // 2

13. Server
In about same time Ryan Dahl creates Node.js, which
implements CommonJS Modules
add.js
exports.add = function() {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
increment.js
var add = require('add').add;
exports.increment = function(val) {
return add(val, 1);
};
program.js
var inc = require('increment').increment;
var a = 1;
inc(a); // 2

14. Server
In about same time Ryan Dahl creates Node.js, which
implements CommonJS Modules
... with some improvements
add.js
module.exports = function() {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
increment.js
var add = require('./add');
module.exports = function(val) {
return add(val, 1);
};
program.js
var inc = require('./increment');
var a = 1;
inc(a); // 2

15. Server
How module works:
// increment.js
var add = require('./add');
module.exports = function (val) {
return add(val, 1);
};

16. Server
How module works:
var exports, module;
function (exports, require, module) {
// increment.js
var add = require('./add');
module.exports = function (val) {
return add(val, 1);
};
}.call(exports = {}, exports, function (path) {
// import external module
}, module = { exports: exports });
MODULE = module.exports;

17. Server
Let’s get back to module pattern.

18. Server
Let’s get back to module pattern.
module.js
MAIN.module = (function() {
var privateVar = '..';
var privateMethod = function (args) {
// ...
};
return {
publicProperty: '..',
publicMethod: function () {
// ...
}
};
}());

19. Server
Module pattern as CommonJS module:
module.js
var privateVar = '..';
var privateMethod = function (args) {
// ...
};
exports.publicProperty: '..',
exports.publicMethod: function () {
// ...
};
program.js
var foobar = require(‘./module’);

20. Server
Module pattern as CommonJS module:
module.js
var privateVar = '..';
var privateMethod = function (args) {
// ...
};
exports.publicProperty: '..',
exports.publicMethod: function () {
// ...
};
program.js
var foobar = require(‘./module’);
↑ we decide locally under which name we’ll access imported module

21. Server
CommonJS Modules is indeed great specification.
What are the benefits ?

22. Server
CommonJS Modules is indeed great specification.
What are the benefits ?
First of all, cleanliness and encapsulation on highest
available (for JavaScript) level

23. Server
CommonJS Modules is indeed great specification.
What are the benefits ?
First of all, cleanliness and encapsulation on highest
available (for JavaScript) level
• There’s no need to write obsolete function wrappers to work in private
scope

24. Server
CommonJS Modules is indeed great specification.
What are the benefits ?
First of all, cleanliness and encapsulation on highest
available (for JavaScript) level
• There’s no need to write obsolete function wrappers to work in private
scope
• We don’t have to deal with long namespaces, each required module is
assigned to local variable

25. Server
CommonJS Modules is indeed great specification.
What are the benefits ?
First of all, cleanliness and encapsulation on highest
available (for JavaScript) level
• There’s no need to write obsolete function wrappers to work in private
scope
• We don’t have to deal with long namespaces, each required module is
assigned to local variable
• We can build large complex application without a need to touch global
namespace

26. Server & Browser
How to run it in the Browser ?

27. Server & Browser
How to run it in the Browser ?
When I started working with CommonJS and Node.js
(beginning 2011) I thought there’s solid solution already
available

28. Server & Browser
How to run it in the Browser ?
When I started working with CommonJS and Node.js
(beginning 2011) I thought there’s solid solution already
available
I’ve found about 8-10 solutions that aimed at something
similar, however none of them were really working right

29. Server & Browser
Main issue was that each of them, was trying to be
something more than Node.js modules parser, and “more”
seemed to have more attention than dependency parser I
was after.

30. Server & Browser
Main issue was that each of them, was trying to be
something more than Node.js modules parser, and “more”
seemed to have more attention than dependency parser I
was after.
Parsers were incomplete and buggy

31. Server & Browser
Anyway, time goes by, there are few solutions that
deserve focus

32. Server & Browser
Anyway, time goes by, there are few solutions that
deserve focus
Browserify -> https://github.com/substack/node-browserify

33. Server & Browser
Anyway, time goes by, there are few solutions that
deserve focus
Browserify -> https://github.com/substack/node-browserify
Very interesting tool, if we’re after reusing native Node.js modules on
client-side. Browserify struggles to make it possible, I guess it was main
intention behind the project.

34. Server & Browser
Anyway, time goes by, there are few solutions that
deserve focus
Browserify -> https://github.com/substack/node-browserify
Very interesting tool, if we’re after reusing native Node.js modules on
client-side. Browserify struggles to make it possible, I guess it was main
intention behind the project.
You need to add extra 320 lines of code for client-side to make it work,
that’s not impressive, this code could be much smaller.

35. Server & Browser
Anyway, time goes by, there are few solutions that
deserve focus
Browserify -> https://github.com/substack/node-browserify
Very interesting tool, if we’re after reusing native Node.js modules on
client-side. Browserify struggles to make it possible, I guess it was main
intention behind the project.
You need to add extra 320 lines of code for client-side to make it work,
that’s not impressive, this code could be much smaller.
It doesn’t work well with some paths, modules from external packages
were not found when I played with that (not sure if it’s bug or limitation)

36. Server & Browser
AMD -> Asynchronous Module Definition

37. Server & Browser
AMD -> Asynchronous Module Definition
It gained some attention lately:
http://addyosmani.com/writing-modular-js/
http://blog.millermedeiros.com/2011/09/amd-is-better-for-the-web-than-commonjs-
modules/
http://unscriptable.com/index.php/2011/09/30/amd-versus-cjs-whats-the-best-
format/

38. Server & Browser
AMD ?

39. Server & Browser
AMD ?
It was made for browsers - Node.js won’t run AMD module properly

40. Server & Browser
AMD ?
It was made for browsers - Node.js won’t run AMD module properly

41. Server & Browser
AMD ?
It was made for browsers - Node.js won’t run AMD module properly
AMD has not much to do with CommonJS Modules, you need to
write modules differently

42. Server & Browser
AMD ?
It was made for browsers - Node.js won’t run AMD module properly
AMD has not much to do with CommonJS Modules, you need to
write modules differently
• All dependencies needs to be declared at begin of module

43. Server & Browser
AMD ?
It was made for browsers - Node.js won’t run AMD module properly
AMD has not much to do with CommonJS Modules, you need to
write modules differently
• All dependencies needs to be declared at begin of module
In CommonJS we can reach for external modules in any place at any time, and it’s
same for native modules that JavaScript will have in a future.

44. Server & Browser
AMD ?
It was made for browsers - Node.js won’t run AMD module properly
AMD has not much to do with CommonJS Modules, you need to
write modules differently
• All dependencies needs to be declared at begin of module
In CommonJS we can reach for external modules in any place at any time, and it’s
same for native modules that JavaScript will have in a future.
• Again, we need to use function wrappers. Each module needs to be declared with
function call

45. Server & Browser
Why is that ?

46. Server & Browser
Why is that ?
AMD tries to be solution for two problems:
• Code modularization and organization
• Dynamic dependency handling, so we load only that
code that is actually used

47. Server & Browser
Why is that ?
AMD tries to be solution for two problems:
• Code modularization and organization
• Dynamic dependency handling, so we load only that
code that is actually used
Sounds promising, but is it worth it ?

48. Server & Browser
How AMD modules look like ?

49. Server & Browser
How AMD modules look like ?
add.js
define(function () {
return function() {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
});
increment.js
define(['add'], function (add) {
return function(val) {
return add(val, 1);
};
});
program.js
define(['increment'], function (inc) {
var a = 1;
inc(a); // 2
});

50. Server & Browser
Most popular implementation -> http://requirejs.org/
requires us to load library made of 2000 lines of code.

51. Server & Browser
Most popular implementation -> http://requirejs.org/
requires us to load library made of 2000 lines of code.
... it’s needed just to run code of our application.

52. Server & Browser
Most popular implementation -> http://requirejs.org/
requires us to load library made of 2000 lines of code.
... it’s needed just to run code of our application.
... and still we cannot use our modules on server-side
(Node.js) without extra compilation step.

53. Server & Browser
Ok, but what about dynamic dependency resolution, isn’t it
main purpose of AMD ?

54. Server & Browser
Ok, but what about dynamic dependency resolution, isn’t it
main purpose of AMD ?
In my opinion dynamic dependency resolution should be
considered on application functionality level not on
modules level

55. Server & Browser
Summary:

56. Server & Browser
Summary:
We miss light and clean CommonJS (precisely Node.js) Modules
implementation, that would allow us to use them on client side

57. Server & Browser
Summary:
We miss light and clean CommonJS (precisely Node.js) Modules
implementation, that would allow us to use them on client side
Implementation that wouldn’t try to be solution for other
problems in first place.

58. Server & Browser
Summary:
We miss light and clean CommonJS (precisely Node.js) Modules
implementation, that would allow us to use them on client side
Implementation that wouldn’t try to be solution for other
problems in first place.
Code for handling modules on client side should be minimal,
unnoticeable in size when compared to application code

59. Server & Browser
Finally new project was born:

60. Server & Browser
Finally new project was born:
modules-webmake ->
https://github.com/medikoo/modules-webmake
npm install -g webmake

61. Modules Webmake
modules-webmake scans modules, builds dependency tree and
produces output that it executable in the browser

62. Modules Webmake
modules-webmake scans modules, builds dependency tree and
produces output that it executable in the browser
Current implementation is very basic, we might say, we just started

63. Modules Webmake
modules-webmake scans modules, builds dependency tree and
produces output that it executable in the browser
Current implementation is very basic, we might say, we just started
It doesn’t limit us though from building complex applications made
from hundreds of modules that originated from dozens of packages

64. Modules Webmake
modules-webmake scans modules, builds dependency tree and
produces output that it executable in the browser
Current implementation is very basic, we might say, we just started
It doesn’t limit us though from building complex applications made
from hundreds of modules that originated from dozens of packages
With modules-webmake you can build complex applications, right
now

65. Modules Webmake
How generated file looks like ?

66. Modules Webmake
How generated file looks like ?
(function (modules) {
// 53 lines of import/export logic
})
({
"root": {
"add.js": function (exports, module, require) {
module.exports = function () {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
},
"increment.js": function (exports, module, require) {
var add = require('./add');
module.exports = function (val) {
return add(val, 1);
};
},
"program.js": function (exports, module, require) {
var inc = require('./increment');
var a = 1;
inc(a); // 2
}
}
})
("root/program");

67. Modules Webmake
How generated file looks like ?
(function (modules) {
// 53 lines of import/export logic
})
({
"root": {
"add.js": function (exports, module, require) {
module.exports = function () {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
},
"increment.js": function (exports, module, require) {
var add = require('./add');
module.exports = function (val) {
return add(val, 1);
};
},
"program.js": function (exports, module, require) {
var inc = require('./increment');
var a = 1;
inc(a); // 2
}
}
})
("root/program"); <- Execution of module that initiates application

68. Modules Webmake
• modules-webmake can be used both from shell and programmatically - we can
bind it to http server in Node.js and generate files on demand

69. Modules Webmake
• modules-webmake can be used both from shell and programmatically - we can
bind it to http server in Node.js and generate files on demand
• Dependencies are parsed statically, it introduces some limitations, for specific
cases we may force inclusion of chosen modules with ‘include’ option.

70. Modules Webmake
• modules-webmake can be used both from shell and programmatically - we can
bind it to http server in Node.js and generate files on demand
• Dependencies are parsed statically, it introduces some limitations, for specific
cases we may force inclusion of chosen modules with ‘include’ option.
• modules-webmake reads local dependencies and those from external packages,
one limitation is that it doesn’t recognize two different versions of same package (it
will be addressed)

71. Modules Webmake
• modules-webmake can be used both from shell and programmatically - we can
bind it to http server in Node.js and generate files on demand
• Dependencies are parsed statically, it introduces some limitations, for specific
cases we may force inclusion of chosen modules with ‘include’ option.
• modules-webmake reads local dependencies and those from external packages,
one limitation is that it doesn’t recognize two different versions of same package (it
will be addressed)
• Up to date documentation can be found at github project page:
https://github.com/medikoo/modules-webmake

72. Modules Webmake
Last week I finished work on application that was built with help of
modules-webmake.
• It’s HTML5 application, targetted only for modern browsers (newest FF, Chrome and
Safari (both OSX & iOS))
• To support Offline mode we needed to put whole application logic to client side,
including templates and simple database engine.
• Node.js on server-side, takes care about clients synchronization and saving data to
physical database (mongodb)
• Minimalistic client-server communication based on sockets (Socket.IO)

73. Modules Webmake
Final JavaScript file for client consists of 273 modules from 19 packages.
176 of mentioned modules works also on server side.
It means that about 60% of client-side code is also running on server-side.
Concatenated file is 11 thousand lines long
and weights about 370kB before minification and zipping.
It looks quite promising, if we take into account, that it consists of templates for
whole application and all other modules that application is made of, including simple
database engine.
After application load, all client-server communication is minimal as all pages as
generated by client.

74. Modules Webmake
Upcoming improvements:

75. Modules Webmake
Upcoming improvements:
• Not only parser, but full modules engine. What for? It will allow to run and test
modules with it’s dependencies in differently setup environments, like pure V8 +
Modules, or emulation of browser environment.

76. Modules Webmake
Upcoming improvements:
• Not only parser, but full modules engine. What for? It will allow to run and test
modules with it’s dependencies in differently setup environments, like pure V8 +
Modules, or emulation of browser environment.
• Work in a background, it will allow much more optimized and faster file generation
(watching file system for changes, immediate serving of generated files when
integrated with node.js http server)

77. Modules Webmake
Upcoming improvements:
• Not only parser, but full modules engine. What for? It will allow to run and test
modules with it’s dependencies in differently setup environments, like pure V8 +
Modules, or emulation of browser environment.
• Work in a background, it will allow much more optimized and faster file generation
(watching file system for changes, immediate serving of generated files when
integrated with node.js http server)
• Optional minification, compilation

78. Modules Webmake
Upcoming improvements:
• Not only parser, but full modules engine. What for? It will allow to run and test
modules with it’s dependencies in differently setup environments, like pure V8 +
Modules, or emulation of browser environment.
• Work in a background, it will allow much more optimized and faster file generation
(watching file system for changes, immediate serving of generated files when
integrated with node.js http server)
• Optional minification, compilation
• Discovering modules that were required but not used

79. Modules Webmake
Upcoming improvements:
• Not only parser, but full modules engine. What for? It will allow to run and test
modules with it’s dependencies in differently setup environments, like pure V8 +
Modules, or emulation of browser environment.
• Work in a background, it will allow much more optimized and faster file generation
(watching file system for changes, immediate serving of generated files when
integrated with node.js http server)
• Optional minification, compilation
• Discovering modules that were required but not used
• Splitting into few files (for faster client-side load)

80. Modules Webmake
Upcoming improvements:
• Not only parser, but full modules engine. What for? It will allow to run and test
modules with it’s dependencies in differently setup environments, like pure V8 +
Modules, or emulation of browser environment.
• Work in a background, it will allow much more optimized and faster file generation
(watching file system for changes, immediate serving of generated files when
integrated with node.js http server)
• Optional minification, compilation
• Discovering modules that were required but not used
• Splitting into few files (for faster client-side load)
• Modules should not be duplicated in different files, introduction of intelligent more
than one file generation

81. Modules Webmake
Upcoming improvements:
• Not only parser, but full modules engine. What for? It will allow to run and test
modules with it’s dependencies in differently setup environments, like pure V8 +
Modules, or emulation of browser environment.
• Work in a background, it will allow much more optimized and faster file generation
(watching file system for changes, immediate serving of generated files when
integrated with node.js http server)
• Optional minification, compilation
• Discovering modules that were required but not used
• Splitting into few files (for faster client-side load)
• Modules should not be duplicated in different files, introduction of intelligent more
than one file generation
• Small code fixes for buggy engines e.g. quoting reserved keywords used as
property names

82. Future: Harmony
We will have modules natively in JavaScript
http://wiki.ecmascript.org/doku.php?id=harmony:modules_examples

83. Future: Harmony
We will have modules natively in JavaScript
http://wiki.ecmascript.org/doku.php?id=harmony:modules_examples
Base concept of Harmony Modules is same as in
CommonJS Modules.

84. Future: Harmony
We will have modules natively in JavaScript
http://wiki.ecmascript.org/doku.php?id=harmony:modules_examples
Base concept of Harmony Modules is same as in
CommonJS Modules.
Differences lies in dedicated syntax, and powerful
optional features like dynamic loading, possibility to load
external modules via url etc.

85. Harmony Modules

86. Harmony Modules
What we currently write for Node.js:
add.js
module.exports = function() {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
increment.js
var add = require('./add');
module.exports = function(val) {
return add(val, 1);
};
program.js
var inc = require('./increment');
var a = 1;
inc(a); // 2

87. Harmony Modules
Will be written that way with Harmony:
add.js
export function add () {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
increment.js
import add from './add';
export function increment (val) {
return add(val, 1);
};
program.js
import { increment: inc } from './increment';
var a = 1;
inc(a); // 2

88. Harmony Modules
Modules Webmake for EcmaScript 3/5:
(function (modules) {
// 53 lines of import/export logic
})
({
"root": {
"add.js": function (exports, module, require) {
module.exports = function () {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
},
"increment.js": function (exports, module, require) {
var add = require('./add');
module.exports = function (val) {
return add(val, 1);
};
},
"program.js": function (exports, module, require) {
var inc = require('./increment');
var a = 1;
inc(a); // 2
}
}
})
("root/program");

89. Harmony Modules
Modules Webmake for Harmony:
module add {
export function add () {
var sum = 0, i = 0, args = arguments, l = args.length;
while (i < l) sum += args[i++];
return sum;
};
};
module increment {
import add from add;
export function increment (val) {
return add(val, 1);
};
};
module program {
import { increment: inc } from increment;
var a = 1;
inc(a); // 2
};
import * from program;

90. Server & Browser
If you use CommonJS/Node.js Modules today, it won’t be
problematic for you to switch to Harmony in few years
time

91. Server & Browser
If you use CommonJS/Node.js Modules today, it won’t be
problematic for you to switch to Harmony in few years
time
Today:
CommonJS/Node.js Modules & modules-webmake

92. Server & Browser
If you use CommonJS/Node.js Modules today, it won’t be
problematic for you to switch to Harmony in few years
time
Today:
CommonJS/Node.js Modules & modules-webmake
Tomorrow:
Harmony Modules

93. Server & Browser
If you use CommonJS/Node.js Modules today, it won’t be
problematic for you to switch to Harmony in few years
time
Today:
CommonJS/Node.js Modules & modules-webmake
Tomorrow:
Harmony Modules (& modules-webmake ?)

94. Questions ?

95. Thank You !
Mariusz Nowak
mariusz@medikoo.com
@medikoo
github.com/medikoo