JavaScript Parser
Infrastructure for
Code Quality Analysis
Ariya Hidayat
Twitter: @AriyaHidayat
http://ariya.ofilabs.com

Who Am I

Do you...
 have a coding style/guidelines?
 use code quality tool such as JSLint or JSHint?
 actively prevent performance & coverage regressions?
 wish you could write your own quality rules?

Code Quality

High Quality: Practical Aspects
Do not provoke ambiguities
Avoid silly mistakes
Learn better code pattern
Write readable code
Improve future maintenance

Better Use of the Tools
Not everyone is
a JavaScript ninja
Engineer
Tools
•Boring
•Repetitive Feedback Cycle
•Time-consuming

From Spelling Checker to Grammar Enforcement Agent
Your so wrong,
therefore you loose!
No misspelled word.
Wrong choice of words!

Adaptive Quality Criteria
Explicit Implicit
Customize analysis options Infer from high-quality sample
Define new sets of rules Observe the engineer’s behavior

Next-Generation Code Quality Tools
To boldly analyze what no man has analyzed before...

Parser Infrastructure

JavaScript in the Browser
User Interface
Data Persistence
Browser Engine
Render Engine
JavaScript Graphics
Networking I/O
Engine Stack

JavaScript Engine Building Blocks
Built-in objects,
Runtime
host objects, ...
Syntax Tree
Virtual
Source Parser Machine/
Interpreter
Fast and conservative

Tokenization
identifier number
var answer = 42;
keyword equal sign
semicolon

Syntax Tree
Variable Declaration
Identifier Literal Constant
answer 42

JavaScript Parser Written in JavaScript
Esprima UglifyJS
Narcissus
ZeParser
Traceur
Es-Lab

Produce Correct Output
 ECMA-262 compliant
'use strict';  Automatic semicolon insertion
var 日本語 = 1
 Strict Mode, e.g. “use strict”
return 日本語
 Unicode for identifiers

Heavily Tested
 > 500 unit tests
 Compatibility tests
 100% code coverage
 Performance tests
Enforced during
development

Sensible Syntax Tree https://developer.mozilla.org/en/SpiderMonkey/Parser_API
{
type: "Program",
body: [
{
type: "ExpressionStatement",
expression: {
answer = 42 type: "AssignmentExpression",
operator: "=",
left: {
type: "Identifier",
name: "answer"
},
right: {
type: "Literal",
value: 42
}
http://esprima.org/demo/parse.html }
}
]
Try online! }

Syntax Tree Visualization
answer = 42

Specification, Parser Code, Syntax Tree
ECMA-262 Annex A.4 while ( Expression ) Statement
function parseWhileStatement() {
var test, body;
expectKeyword('while');
expect('(');
test = parseExpression();
expect(')');
body = parseStatement();
return {
type: Syntax.WhileStatement,
test: test,
body: body
};
}

High Performance http://esprima.org/test/compare.html
Speed Comparison
Chrome 18
Internet Explorer 9
Esprima 233 ms 567 ms
UglifyJS parse-js 620 ms 922 ms
Benchmark corpus:
jQuery, Prototype, MooTools, ExtCore, ...

Syntax Node Location
{
type: "ExpressionStatement",
expression: {
type: "AssignmentExpression",
operator: "=",
left: {
answer = 42
type: "Identifier",
name: "answer",
range: [0, 6]
},
right: {
type: "Literal",
value: 42,
range: [9, 11]
},
range: [0, 11]
},
range: [0, 11]
}

Fit for Code Regeneration https://github.com/Constellation/escodegen
Source Esprima Escodegen Source
Syntax Tree
Shorten variable name
Syntax Transformation Automatically inline short function
Obfuscate

Easily Tweaked LLJS: Low Level JavaScript http://mbebenita.github.com/LLJS/
Block scope let x = 0;
let u8 flag;
Data types
let i32 position;
struct Point {
int x, y;
};
Pointers let u16 *p = q;

Error Tolerant Useful for IDE, editors, ...
Mismatched quote var msg = "Hello’;
Too many dots person..age = 18;
Incomplete, still typing if (person.
Strict mode violation 'use strict';
with (person) {
}

Handle the Comments
// Life, Universe, and Everything
answer = 42
comments: [
{
range: [0, 34],
type: "Line",
value: " Life, Universe, and Everything"
}
]
Documentation tool Code annotation
https://github.com/thejohnfreeman/jfdoc https://github.com/goatslacker/node-typedjs

Forward Looking Experimental ‘harmony’ branch
Object initializer shorthand Destructuring assignment
var point = {x, y}; point = {14, 3, 77};
{x, y, z} = point;
Module
Block scope
module MathExtra {
// do something {
let x;
} const y = 0;
}

Code Quality Tools

Syntax Tree Visualization
Block statement
http://esprima.org/demo/parse.html

Most Popular Keywords
this 3229
function 3108
if 3063
return 2878
var 2116 var fs = require('fs'),
else 562
esprima = require('esprima'),
for 436
files = process.argv.splice(2);
new 232
in 225
files.forEach(function (filename) {
typeof 188
var content = fs.readFileSync(filename, 'utf-8'),
while 143
tokens = esprima.parse(content, { tokens: true }).tokens;
case 122
break 115
tokens.forEach(function (token) {
try 84
if (token.type === 'Keyword') {
catch 84
console.log(token.value);
delete 72
}
throw 38
});
switch 35
});
continue 25
default 15
instanceof 14
do 12
void 10
finally 4
http://ariya.ofilabs.com/2012/03/most-popular-javascript-keywords.html

Most Popular Statements
var fs = require('fs'),
ExpressionStatement 6728
esprima = require('esprima'),
BlockStatement 6353 files = process.argv.splice(2);
IfStatement 3063
ReturnStatement 2878 files.forEach(function (filename) {
VariableDeclaration 2116 var content = fs.readFileSync(filename, 'utf-8'),
syntax = esprima.parse(content);
FunctionDeclaration 371
ForStatement 293
JSON.stringify(syntax, function (key, value) {
ForInStatement 143
if (key === 'type') {
WhileStatement 131
if (value.match(/Declaration$/) ||
BreakStatement 115 value.match(/Statement$/)) {
TryStatement 84 console.log(value);
EmptyStatement 66 }
}
ThrowStatement 38
return value;
SwitchStatement 35
});
ContinueStatement 25
});
DoWhileStatement 12
LabeledStatement 6
http://ariya.ofilabs.com/2012/04/most-popular-javascript-statements.html

Identifier Length Distribution
750 mean of the identifier length is 8.27 characters
500
prototype-1.7.0.0.js SCRIPT_ELEMENT_REJECTS_TEXTNODE_APPENDING
prototype-1.7.0.0.js MOUSEENTER_MOUSELEAVE_EVENTS_SUPPORTED
jquery-1.7.1.js subtractsBorderForOverflowNotVisible
jquery.mobile-1.0.js getClosestElementWithVirtualBinding
250 prototype-1.7.0.0.js HAS_EXTENDED_CREATE_ELEMENT_SYNTAX
0
0 5 10 15 20 25 30 35 40 45
http://ariya.ofilabs.com/2012/05/javascript-identifier-length-distribution.html

“Code Linting”
var fs = require('fs'),
esprima = require('./esprima'),
files = process.argv.splice(2);
Not a strict equal files.forEach(function (filename) {
var content = fs.readFileSync(filename, 'utf-8'),
syntax = esprima.parse(content, { loc: true });
if (x == 9) {
// do Something JSON.stringify(syntax, function (key, value) {
} if (key === 'test' && value.operator === '==')
console.log('Line', value.loc.start.line);
return value;
});
});
Another example: next-generation JSHint https://github.com/jshint/jshint-next

Strict Mode Checking
Duplicate data property in object
literal not allowed in strict mode
'use strict';
block = {
color: 'blue',
height: 20,
width: 10,
color: 'red'
};
http://ariya.ofilabs.com/2012/03/strict-mode-checks-with-esprima.html

http://ariya.ofilabs.com/2011/08/hall-of-api-shame-boolean-trap.html
“Boolean Trap” Finder
var volumeSlider = new Slider(false);
Obfuscated choice
component.setHidden(false);
Double-negative filter.setCaseInsensitive(false);
Can you make up your mind? treeItem.setState(true, false);
event.initKeyEvent("keypress", true, true, null, null,
The more the merrier? false, false, false, false, 9, 0);

Style Formatter https://github.com/fawek/codepainter
Sample code
Source CodePainter Formatted source
Infer coding styles Indentation
Quote for string literal
Whitespace

https://github.com/Constellation/escodegen
Rewrite and Regenerate
answer = 42;
var syntax = esprima.parse('answer = 42;');
syntax.body[0].expression.right.value = 1337;
escodegen.generate(syntax)
answer = 1337;

Code Coverage
https://github.com/itay/node-cover
https://github.com/coveraje/coveraje
x = 42;
https://github.com/pmlopes/coberturajs
if (false)
x = -1;
http://ariya.ofilabs.com/2012/03/javascript-code-coverage-and-esprima.html

http://itay.github.com/snug_codecoverage_slides/
Instrumentation for Coverage
{
__statement_ZyyqFc(1);
var a = 5;
Statement var a = 5;
}
{
__statement_ZyyqFc(2);
Expression foo(); __expression_kC$jur(3), foo();
}
function foo() { function foo() {
Block ... __block_n53cJc(1);
}; ...
}

Non-Destructive Partial Source Modification
Do not remove comments
Intact
Preserve indentation & other formatting
Modified Add “contextual” information
Inject or change function invocation

String Literal Quotes May need proper escaping
console.log("Hello") console.log('Hello')
List of tokens
[
{ type: "Identifier", value: "console", range: [0, 7] },
{ type: "Punctuator", value: ".", range: [7, 8] },
{ type: "Identifier", value: "log", range: [8, 11] },
{ type: "Punctuator", value: "(", range: [11, 12] },
{ type: "String", value: ""Hello"", range: [12, 19] },
{ type: "Punctuator", value: ")", range: [19, 19] }
]
http://ariya.ofilabs.com/2012/02/from-double-quotes-to-single-quotes.html

Tracking the Scalability
Array.prototype.swap = function (i, j) {
var k = this[i]; this[i] = this[j]; this[j] = k;
}
Array.prototype.swap = function (i, j) {
Log({ name: 'Array.prototype.swap', lineNumber: 1, range: [23, 94] });
var k = this[i]; this[i] = this[j]; this[j] = k;
}
http://ariya.ofilabs.com/2012/01/scalable-web-apps-the-complexity-issue.html

Execution Tracing 4640 function calls
jQuery Mobile startup log https://gist.github.com/1823129
jquery.js 26 jQuery
jquery.js 103 init undefined, undefined, [object Object]
jquery.js 274 each (Function)
jquery.js 631 each [object Object], (Function), undefined
jquery.js 495 isFunction [object Object]
jquery.js 512 type [object Object]
jquery.mobile.js 1857 [Anonymous]
jquery.mobile.js 642 [Anonymous]
jquery.mobile.js 624 enableMouseBindings
jquery.mobile.js 620 disableTouchBindings
http://ariya.ofilabs.com/2012/02/tracking-javascript-execution-during-startup.html

http://esprima.googlecode.com/git-history/harmony/demo/transpile.html
Transpilation
// Object literal property shorthand.
function Point(x, y) {
Harmony return { x, y };
}
Intact
// Object literal property shorthand.
ES 5.1 function Point(x, y) {
return { x: x, y: y };
}
Inserted fragment

Application Structure
Class manifest
Ext.define('My.sample.Person', {
name: 'Mr. Unknown',
age: 42,
{
className: 'My.sample.Person',
constructor: function(name) {},
functions: ['walk', 'run'],
properties: ['name', 'age']
walk: function(steps) {}
}
run: function(steps) {}
});
http://www.sencha.com/learn/sencha-class-system/

Code Outline Eclipse
Functions
Variables

Content Assist (aka Autocomplete, aka “Intellisense”)
Eclipse
http://contraptionsforprogramming.blogspot.com/2012/02/better-javascript-content-assist-in.html

Copy Paste Mistake
function inside(point, rect) {
return (point.x >= rect.x1) && (point.y >= rect.y1) &&
(point.x <= rect.x2) && (point.y <= rect.y1);
}
Wrong check

Refactoring Helper
// Add two numbers // Add two numbers
function add(firt, two) { function add(first, two) {
return firt + two; return first + two;
} }

Future
Tree traversal
Scope analysis
“Richer” syntax tree
Syntax transformation
Pattern Matching

Conclusion

Smart editing
Source transformation
Instrumentation
Modern Minification & obfuscation
Parser
Code coverage
Documentation generator
Conditional contracts
Dependency analysis

Thank You!
ariya.hidayat@gmail.com
ariya.ofilabs.com
@AriyaHidayat