The document discusses working with code abstract syntax trees (ASTs). It provides examples of parsing code into ASTs using libraries like Esprima, querying ASTs using libraries like grasp-equery, constructing and transforming ASTs, and generating code from ASTs. It introduces aster, an AST-based code builder that allows defining reusable AST transformations as plugins and integrating AST-based builds into generic build systems like Grunt and Gulp. Aster aims to improve on file-based builders by working directly with ASTs in a streaming fashion.

1. AST (Abstract Syntax Tree)
The only true tool for building JavaScript

2. Source maps
Epic win in debugging

3. Source maps – epic win in debugging

4. Source maps – epic win in debugging

5. Builders
Epic fail in debugging

6. Builders – epic fail in debugging
umdify:
// UMD definition
output += '(function(root, factory) {n';
output += ' if (typeof define === "function" && define.amd) {n';
output += ' define([' + depNames.join(', ') + '], factory);n';
output += ' } else if (typeof exports === "object") {n';
output += ' module.exports = factory(require);n';
…

7. Builders – epic fail in debugging
grunt-amd-wrap:
var srcText = grunt.file.read(file.src[0]);
var destText = amdWrap(srcText);

8. Builders – epic fail in debugging
gulp-concat:
buffer.push(file.contents);
…
var joinedContents = Buffer.concat(buffer);

9. Builders – epic fail in debugging
universal-transformer:
function transform(srcText) {
return 'var answer = 42;';
}

10. Your code is not a string
It has a soul

11. Your code has a soul
// Life, Universe, and Everything
var answer = 6 * 7;

12. Your code has a soul
// Life, Universe, and Everything
var answer = 6 * 7;
'// Life, Universe and
Everythingnvar answer
= 6 * 7;'

13. Your code has a soul
// Life, Universe, and Everything
var answer = 6 * 7;
[
{ type: "Keyword", value: "var" },
{ type: "Identifier", value: "answer" },
{ type: "Punctuator", value: "=" },
{ type: "Numeric", value: "6" },
{ type: "Punctuator", value: "*" },
{ type: "Numeric", value: "7" },
{ type: "Punctuator", value: ";" }
]

14. Your code has a soul
[
{ type: "Keyword", value: "var" },
{ type: "Identifier", value: "answer" },
{ type: "Punctuator", value: "=" },
{ type: "Numeric", value: "6" },
{ type: "Punctuator", value: "*" },
{ type: "Numeric", value: "7" },
{ type: "Punctuator", value: ";" }
]
{
type: "Program",
body: [{
type: "VariableDeclaration",
declarations: [{
type: "VariableDeclarator",
id: {type: "Identifier", name: "answer"},
init: {
type: "BinaryExpression",
operator: "*",
left: {type: "Literal", value: 6},
right: {type: "Literal", value: 7}
}
}],
kind: "var"
}]
}

15. Your code has a soul
{
type: "Program",
body: [{
type: "VariableDeclaration",
declarations: [{
type: "VariableDeclarator",
id: {type: "Identifier", name: "answer"},
init: {
type: "BinaryExpression",
operator: "*",
left: {type: "Literal", value: 6},
right: {type: "Literal", value: 7}
}
}],
kind: "var"
}]
}
Program
VariableDeclaration
VariableDeclarator
Identifier(“answer”) BinaryExpression(*)
Literal(6) Literal(7)

16. Your code has a soul
// Life, Universe, and Everything
var answer = 6 * 7; Program
VariableDeclaration
VariableDeclarator
Identifier(“answer”) BinaryExpression(*)
Literal(6) Literal(7)

17. Code tools
How can we work with code AST?

18. Parsing
• JavaScript
• SpiderMonkey: Reflect.parse – Mozilla's Parser API
• Esprima – most popular ECMAScript parser in JS
• Acorn – faster alternative ECMAScript parser in JS
• UglifyJS – has own parser with custom AST format
• Traceur – has ES6 parser that can be used separately as well
• … (as a lot of language tools do) …
• CoffeeScript
• CoffeeScriptRedux – rewrite of CS compiler that internally uses CS AST with conversion to JS
AST
• JSX
• esprima-fb – Facebook's fork of Esprima Harmony branch
• jsx-esprima – es* tools based JSX to JS AST transpiler

19. Parsing
acorn.parse('var answer = 6 * 7;', {locations: true});
// In each node.
loc: {
start: {
line: 2,
column: 0
},
end: {
line: 2,
column: 19
}
}

20. Linting

21. Querying
var found;
estraverse.traverse(tree, {
enter: function (node) {
if (node.type === 'Identifier' && node.name[0] === '_') {
found = node;
return estraverse.VisitorOption.Break;
}
}
})

22. Querying
require('grasp-equery')
.query('if ($cond) return $yes; else return $no;', ast)

23. Querying
require('grasp-squery')
.query('if[then=return][else=return]', ast)

24. Constructing
{
type: "Program",
body: [{
type: "VariableDeclaration",
declarations: [{
type: "VariableDeclarator",
id: {type: "Identifier", name: "answer"},
init: {
type: "BinaryExpression",
operator: "*",
left: {type: "Literal", value: 6},
right: {type: "Literal", value: 7}
}
}],
kind: "var"
}]
}

25. Constructing
var b = require('ast-types').builders;
b.variableDeclaration('var', [
b.variableDeclarator(
b.identifier('answer'),
b.binaryExpression(
'*',
b.literal(6),
b.literal(7)
)
)
]);

26. Constructing
estemplate('var <%= varName %> = 6 * 7;', {
varName: {type: 'Identifier', name: 'answer'}
});

27. Transforming
var counter = 0, map = Object.create(null);
result = estraverse.replace(tree, {
enter: function (node) {
if (node.type === 'Identifier' && node.name[0] === '_')
node.name = map[node.name] || (map[node.name] = '$' + counter++);
}
});

28. Transforming
var Renamer = recast.Visitor.extend({
init: function () {
this.counter = 0;
this.map = Object.create(null);
},
getId: function (name) {
return this.map[name] || (this.map[name] = '$' + this.counter++);
},
visitIdentifier: function (node) {
if (node.name[0] === '_') node.name = this.getId(node.name);
}
});

29. Generating
var output = escodegen.generate(ast, {
sourceMap: true,
sourceMapWithCode: true
});
fs.writeFileSync('out.js', output.code);
fs.writeFileSync('out.js.map', output.map.toString());

30. Building with AST
What can we improve here?

31. File-based builders (Grunt)
Parsing
code
Transforming
AST
Generating
code
Reading
file
Writing file
Plugin

32. Streaming builders (Gulp, Browserify)
Transforming
AST
Generating
code
Parsing code
Reading
file
Writing
file
Plugin

33. Transforming
AST
Next logical step
Reading
file
Writing
file
Parsing
code
Generating
code

34. aster - AST-based code builder
https://github.com/asterjs/aster

35. Already available
aster
src
watch
dest
runner
aster-parse
js
esnext
jsx
coffee
aster-changed
aster-concat
aster-equery
aster-squery
aster-rename-ids
aster-traverse
aster-uglify
aster-umd
npm keyword: aster-plugin

36. Sample build script
aster.watch(['src/**/*.js', 'src/**/*.coffee', 'src/**/*.jsx'])
.throttle(500)
.map(changed(
src => src.map(equery({
'if ($cond) return $expr1; else return $expr2;':
'return <%= cond %> ? <%= expr1 %> : <%= expr2 %>'
}))
))
.map(concat('built.js'))
.map(umd({exports: 'superLib'}))
.map(aster.dest('dist', {sourceMap: true}))
.subscribe(aster.runner);

37. Plugins – reactive AST transformers
module.exports = source => {
source = source || 'built.js';
return files => files
.flatMap(file => Rx.Observable.fromArray(file.program.body))
.toArray()
.map(body => ({
type: 'File',
program: {type: 'Program', body},
loc: {source}
}));
};

38. Integration with generic build systems
grunt.initConfig({
aster: {
options: {
equery: {
'if ($cond) return $expr1; else return $expr2;':
'return <%= cond %> ? <%= expr1 %> : <%= expr2 %>'
},
concat: 'built.js',
umd: {exports: 'superLib'},
dest: {sourceMap: true}
},
files: { 'dist': ['src/**/*.js', 'src/**/*.coffee', 'src/**/*.jsx'] }
}
});

39. Integration with generic build systems
gulp.src(['src/**/*.js', 'src/**/*.coffee', 'src/**/*.jsx'])
.pipe(aster(src => src
.map(equery({
'if ($cond) return $expr1; else return $expr2;':
'return <%= cond %> ? <%= expr1 %> : <%= expr2 %>'
})
.map(concat('built.js'))
.map(umd({exports: 'superLib'}))
))
.pipe(gulp.dest('dist'))

40. Questions?
https://github.com/asterjs
@RReverser
me@rreverser.com