JavaScript Parser Infrastructure for Code Quality Analysis

JavaScript Parser
Infrastructure for
Code Quality Analysis
Ariya Hidayat

Twitter: @AriyaHidayat
http://ariya.ofilabs.com

Do you...

 have a coding style/guidelines?
 use code quality tool such as JSLint or JSHint?
 actively prevent performance & coverage regressions?
 wish you could write your own quality rules?

High Quality: Practical Aspects

Do not provoke ambiguities
Avoid silly mistakes
Learn better code pattern
Write readable code

Improve future maintenance

Better Use of the Tools
Not everyone is
a JavaScript ninja

Engineer
Tools

•Boring
•Repetitive Feedback Cycle
•Time-consuming

From Spelling Checker to Grammar Enforcement Agent

Your so wrong,
therefore you loose!

No misspelled word.
Wrong choice of words!

Adaptive Quality Criteria

Explicit Implicit

Customize analysis options Infer from high-quality sample
Deﬁne new sets of rules Observe the engineer’s behavior

Next-Generation Code Quality Tools

To boldly analyze what no man has analyzed before...

JavaScript in the Browser

User Interface

Data Persistence
Browser Engine

Render Engine

JavaScript Graphics
Networking I/O
Engine Stack

JavaScript Engine Building Blocks

Built-in objects,
Runtime
host objects, ...

Syntax Tree

Virtual
Source Parser Machine/
Interpreter
Fast and conservative

Tokenization

identiﬁer number

var answer = 42;
keyword equal sign
semicolon

Syntax Tree

Variable Declaration

Identiﬁer Literal Constant

answer 42

JavaScript Parser Written in JavaScript

Esprima UglifyJS
Narcissus
ZeParser
Traceur
Es-Lab

Produce Correct Output

 ECMA-262 compliant
'use strict';  Automatic semicolon insertion
var 日本語 = 1
 Strict Mode, e.g. “use strict”
return 日本語
 Unicode for identifiers

Heavily Tested
 > 500 unit tests
 Compatibility tests
 100% code coverage
 Performance tests

Enforced during
development

Sensible Syntax Tree https://developer.mozilla.org/en/SpiderMonkey/Parser_API

{
type: "Program",
body: [
{
type: "ExpressionStatement",
expression: {
answer = 42 type: "AssignmentExpression",
operator: "=",
left: {
type: "Identifier",
name: "answer"
},
right: {
type: "Literal",
value: 42
}
http://esprima.org/demo/parse.html }
}
]
Try online! }

Syntax Tree Visualization

answer = 42

Specification, Parser Code, Syntax Tree
ECMA-262 Annex A.4 while ( Expression ) Statement

function parseWhileStatement() {
var test, body;

expectKeyword('while');
expect('(');
test = parseExpression();
expect(')');
body = parseStatement();

return {
type: Syntax.WhileStatement,
test: test,
body: body
};
}

High Performance http://esprima.org/test/compare.html

Speed Comparison
Chrome 18
Internet Explorer 9
Esprima 233 ms 567 ms

UglifyJS parse-js 620 ms 922 ms

Benchmark corpus:
jQuery, Prototype, MooTools, ExtCore, ...

Syntax Node Location
{
type: "ExpressionStatement",
expression: {
type: "AssignmentExpression",
operator: "=",
left: {
answer = 42
type: "Identifier",
name: "answer",
range: [0, 6]
},
right: {
type: "Literal",
value: 42,
range: [9, 11]
},
range: [0, 11]
},
range: [0, 11]
}

Fit for Code Regeneration https://github.com/Constellation/escodegen

Source Esprima Escodegen Source

Syntax Tree

Shorten variable name
Syntax Transformation Automatically inline short function
Obfuscate

Easily Tweaked LLJS: Low Level JavaScript http://mbebenita.github.com/LLJS/

Block scope let x = 0;

let u8 flag;
Data types
let i32 position;

struct Point {
int x, y;
};

Pointers let u16 *p = q;

Error Tolerant Useful for IDE, editors, ...

Mismatched quote var msg = "Hello’;

Too many dots person..age = 18;

Incomplete, still typing if (person.

Strict mode violation 'use strict';
with (person) {
}

Handle the Comments
// Life, Universe, and Everything
answer = 42

comments: [
{
range: [0, 34],
type: "Line",
value: " Life, Universe, and Everything"
}
]

Documentation tool Code annotation
https://github.com/thejohnfreeman/jfdoc https://github.com/goatslacker/node-typedjs

Forward Looking Experimental ‘harmony’ branch

Object initializer shorthand Destructuring assignment

var point = {x, y}; point = {14, 3, 77};
{x, y, z} = point;

Module
Block scope
module MathExtra {

// do something {
let x;
} const y = 0;
}

Syntax Tree Visualization

Block statement

http://esprima.org/demo/parse.html

Most Popular Keywords
this 3229

function 3108

if 3063

return 2878

var 2116 var fs = require('fs'),
else 562
esprima = require('esprima'),
for 436
files = process.argv.splice(2);
new 232

in 225
files.forEach(function (filename) {
typeof 188
var content = fs.readFileSync(filename, 'utf-8'),
while 143
tokens = esprima.parse(content, { tokens: true }).tokens;
case 122

break 115
tokens.forEach(function (token) {
try 84
if (token.type === 'Keyword') {
catch 84
console.log(token.value);
delete 72
}
throw 38
});
switch 35
});
continue 25

default 15

instanceof 14

do 12

void 10

finally 4
http://ariya.oﬁlabs.com/2012/03/most-popular-javascript-keywords.html

Most Popular Statements
var fs = require('fs'),
ExpressionStatement 6728
esprima = require('esprima'),
BlockStatement 6353 files = process.argv.splice(2);
IfStatement 3063
ReturnStatement 2878 files.forEach(function (filename) {
VariableDeclaration 2116 var content = fs.readFileSync(filename, 'utf-8'),
syntax = esprima.parse(content);
FunctionDeclaration 371

ForStatement 293
JSON.stringify(syntax, function (key, value) {
ForInStatement 143
if (key === 'type') {
WhileStatement 131
if (value.match(/Declaration$/) ||
BreakStatement 115 value.match(/Statement$/)) {
TryStatement 84 console.log(value);
EmptyStatement 66 }
}
ThrowStatement 38
return value;
SwitchStatement 35
});
ContinueStatement 25
});
DoWhileStatement 12

LabeledStatement 6

http://ariya.oﬁlabs.com/2012/04/most-popular-javascript-statements.html

Identifier Length Distribution

750 mean of the identifier length is 8.27 characters

500
prototype-1.7.0.0.js SCRIPT_ELEMENT_REJECTS_TEXTNODE_APPENDING
prototype-1.7.0.0.js MOUSEENTER_MOUSELEAVE_EVENTS_SUPPORTED
jquery-1.7.1.js subtractsBorderForOverflowNotVisible
jquery.mobile-1.0.js getClosestElementWithVirtualBinding
250 prototype-1.7.0.0.js HAS_EXTENDED_CREATE_ELEMENT_SYNTAX

0
0 5 10 15 20 25 30 35 40 45

http://ariya.ofilabs.com/2012/05/javascript-identifier-length-distribution.html

“Code Linting”
var fs = require('fs'),
esprima = require('./esprima'),
files = process.argv.splice(2);

Not a strict equal files.forEach(function (filename) {
var content = fs.readFileSync(filename, 'utf-8'),
syntax = esprima.parse(content, { loc: true });
if (x == 9) {
// do Something JSON.stringify(syntax, function (key, value) {
} if (key === 'test' && value.operator === '==')
console.log('Line', value.loc.start.line);
return value;
});
});

Another example: next-generation JSHint https://github.com/jshint/jshint-next

Strict Mode Checking
Duplicate data property in object
literal not allowed in strict mode

'use strict';
block = {
color: 'blue',
height: 20,
width: 10,
color: 'red'
};

http://ariya.oﬁlabs.com/2012/03/strict-mode-checks-with-esprima.html

http://ariya.oﬁlabs.com/2011/08/hall-of-api-shame-boolean-trap.html
“Boolean Trap” Finder

var volumeSlider = new Slider(false);
Obfuscated choice

component.setHidden(false);
Double-negative filter.setCaseInsensitive(false);

Can you make up your mind? treeItem.setState(true, false);

event.initKeyEvent("keypress", true, true, null, null,
The more the merrier? false, false, false, false, 9, 0);

Style Formatter https://github.com/fawek/codepainter

Sample code

Source CodePainter Formatted source

Infer coding styles Indentation
Quote for string literal
Whitespace

https://github.com/Constellation/escodegen
Rewrite and Regenerate
answer = 42;

var syntax = esprima.parse('answer = 42;');
syntax.body[0].expression.right.value = 1337;
escodegen.generate(syntax)

answer = 1337;

Code Coverage
https://github.com/itay/node-cover
https://github.com/coveraje/coveraje
x = 42;
https://github.com/pmlopes/coberturajs
if (false)
x = -1;

http://ariya.oﬁlabs.com/2012/03/javascript-code-coverage-and-esprima.html

http://itay.github.com/snug_codecoverage_slides/
Instrumentation for Coverage

{
__statement_ZyyqFc(1);
var a = 5;
Statement var a = 5;
}

{
__statement_ZyyqFc(2);
Expression foo(); __expression_kC$jur(3), foo();
}

function foo() { function foo() {
Block ... __block_n53cJc(1);
}; ...
}

Non-Destructive Partial Source Modification

Do not remove comments
Intact
Preserve indentation & other formatting

Modiﬁed Add “contextual” information
Inject or change function invocation

String Literal Quotes May need proper escaping

console.log("Hello") console.log('Hello')

List of tokens
[
{ type: "Identifier", value: "console", range: [0, 7] },
{ type: "Punctuator", value: ".", range: [7, 8] },
{ type: "Identifier", value: "log", range: [8, 11] },
{ type: "Punctuator", value: "(", range: [11, 12] },
{ type: "String", value: ""Hello"", range: [12, 19] },
{ type: "Punctuator", value: ")", range: [19, 19] }
]

http://ariya.oﬁlabs.com/2012/02/from-double-quotes-to-single-quotes.html

Tracking the Scalability

Array.prototype.swap = function (i, j) {
var k = this[i]; this[i] = this[j]; this[j] = k;
}

Array.prototype.swap = function (i, j) {
Log({ name: 'Array.prototype.swap', lineNumber: 1, range: [23, 94] });
var k = this[i]; this[i] = this[j]; this[j] = k;
}

http://ariya.oﬁlabs.com/2012/01/scalable-web-apps-the-complexity-issue.html

Execution Tracing 4640 function calls

jQuery Mobile startup log https://gist.github.com/1823129

jquery.js 26 jQuery
jquery.js 103 init undefined, undefined, [object Object]
jquery.js 274 each (Function)
jquery.js 631 each [object Object], (Function), undefined
jquery.js 495 isFunction [object Object]
jquery.js 512 type [object Object]
jquery.mobile.js 1857 [Anonymous]
jquery.mobile.js 642 [Anonymous]
jquery.mobile.js 624 enableMouseBindings
jquery.mobile.js 620 disableTouchBindings

http://ariya.oﬁlabs.com/2012/02/tracking-javascript-execution-during-startup.html

http://esprima.googlecode.com/git-history/harmony/demo/transpile.html
Transpilation

// Object literal property shorthand.
function Point(x, y) {
Harmony return { x, y };
}

Intact
// Object literal property shorthand.
ES 5.1 function Point(x, y) {
return { x: x, y: y };
}

Inserted fragment

Application Structure

Class manifest
Ext.define('My.sample.Person', {
name: 'Mr. Unknown',
age: 42,
{
className: 'My.sample.Person',
constructor: function(name) {},
functions: ['walk', 'run'],

properties: ['name', 'age']
walk: function(steps) {}
}
run: function(steps) {}

});

http://www.sencha.com/learn/sencha-class-system/

Code Outline Eclipse

Functions
Variables

Content Assist (aka Autocomplete, aka “Intellisense”)

Eclipse

http://contraptionsforprogramming.blogspot.com/2012/02/better-javascript-content-assist-in.html

Copy Paste Mistake

function inside(point, rect) {
return (point.x >= rect.x1) && (point.y >= rect.y1) &&
(point.x <= rect.x2) && (point.y <= rect.y1);
}
Wrong check

Refactoring Helper

// Add two numbers // Add two numbers
function add(firt, two) { function add(first, two) {
return firt + two; return first + two;
} }

Future

Tree traversal
Scope analysis
“Richer” syntax tree
Syntax transformation
Pattern Matching

Smart editing
Source transformation
Instrumentation

Modern Miniﬁcation & obfuscation
Parser
Code coverage

Documentation generator

Conditional contracts
Dependency analysis

Thank You!

ariya.hidayat@gmail.com

ariya.oﬁlabs.com

@AriyaHidayat

JavaScript Parser Infrastructure for Code Quality Analysis

More Related Content

JavaScript Parser Infrastructure for Code Quality Analysis