JavaScript Parser Infrastructure for Code Quality Analysis

JavaScript Parser Infrastructure forCode Quality Analysis Ariya Hidayat Twitter: @AriyaHidayat http://ariya.ofilabs.com

Who Am I

Do you...  have a coding style/guidelines?  use code quality tool such as JSLint or JSHint?  actively prevent performance & coverage regressions?  wish you could write your own quality rules?

Code Quality

High Quality: Practical Aspects Do not provoke ambiguities Avoid silly mistakes Learn better code pattern Write readable code Improve future maintenance

Better Use of the Tools Not everyone is a JavaScript ninja Engineer Tools •Boring •Repetitive Feedback Cycle •Time-consuming

From Spelling Checker to Grammar Enforcement Agent Your so wrong, therefore you loose! No misspelled word. Wrong choice of words!

Adaptive Quality Criteria Explicit ImplicitCustomize analysis options Infer from high-quality sampleDeﬁne new sets of rules Observe the engineer’s behavior

Next-Generation Code Quality Tools To boldly analyze what no man has analyzed before...

Parser Infrastructure

JavaScript in the Browser User Interface Data Persistence Browser Engine Render Engine JavaScript Graphics Networking I/O Engine Stack

JavaScript Engine Building Blocks Built-in objects, Runtime host objects, ... Syntax Tree Virtual Source Parser Machine/ Interpreter Fast and conservative

Tokenization identiﬁer number var answer = 42; keyword equal sign semicolon

Syntax Tree Variable Declaration Identiﬁer Literal Constant answer 42

JavaScript Parser Written in JavaScript Esprima UglifyJS Narcissus ZeParser Traceur Es-Lab

Produce Correct Output  ECMA-262 compliant use strict;  Automatic semicolon insertion var 日本語 = 1  Strict Mode, e.g. “use strict” return 日本語  Unicode for identifiers

Heavily Tested  > 500 unit tests  Compatibility tests  100% code coverage  Performance tests Enforced during development

Sensible Syntax Tree https://developer.mozilla.org/en/SpiderMonkey/Parser_API { type: "Program", body: [ { type: "ExpressionStatement", expression: { answer = 42 type: "AssignmentExpression", operator: "=", left: { type: "Identifier", name: "answer" }, right: { type: "Literal", value: 42 } http://esprima.org/demo/parse.html } } ] Try online! }

Syntax Tree Visualization answer = 42

Specification, Parser Code, Syntax Tree ECMA-262 Annex A.4 while ( Expression ) Statement function parseWhileStatement() { var test, body; expectKeyword(while); expect((); test = parseExpression(); expect()); body = parseStatement(); return { type: Syntax.WhileStatement, test: test, body: body }; }

High Performance http://esprima.org/test/compare.html Speed Comparison Chrome 18 Internet Explorer 9 Esprima 233 ms 567 ms UglifyJS parse-js 620 ms 922 ms Benchmark corpus: jQuery, Prototype, MooTools, ExtCore, ...

Syntax Node Location { type: "ExpressionStatement", expression: { type: "AssignmentExpression", operator: "=", left: { answer = 42 type: "Identifier", name: "answer", range: [0, 6] }, right: { type: "Literal", value: 42, range: [9, 11] }, range: [0, 11] }, range: [0, 11] }

Fit for Code Regeneration https://github.com/Constellation/escodegen Source Esprima Escodegen Source Syntax Tree Shorten variable name Syntax Transformation Automatically inline short function Obfuscate

Easily Tweaked LLJS: Low Level JavaScript http://mbebenita.github.com/LLJS/ Block scope let x = 0; let u8 flag; Data types let i32 position; struct Point { int x, y; }; Pointers let u16 *p = q;

Error Tolerant Useful for IDE, editors, ... Mismatched quote var msg = "Hello’; Too many dots person..age = 18; Incomplete, still typing if (person. Strict mode violation use strict; with (person) { }

Handle the Comments // Life, Universe, and Everything answer = 42 comments: [ { range: [0, 34], type: "Line", value: " Life, Universe, and Everything" } ] Documentation tool Code annotation https://github.com/thejohnfreeman/jfdoc https://github.com/goatslacker/node-typedjs

Forward Looking Experimental ‘harmony’ branch Object initializer shorthand Destructuring assignment var point = {x, y}; point = {14, 3, 77}; {x, y, z} = point; Module Block scope module MathExtra { // do something { let x; } const y = 0; }

Code Quality Tools

Syntax Tree Visualization Block statement http://esprima.org/demo/parse.html

Most Popular Keywords this 3229 function 3108 if 3063 return 2878 var 2116 var fs = require(fs), else 562 esprima = require(esprima), for 436 files = process.argv.splice(2); new 232 in 225 files.forEach(function (filename) { typeof 188 var content = fs.readFileSync(filename, utf-8), while 143 tokens = esprima.parse(content, { tokens: true }).tokens; case 122 break 115 tokens.forEach(function (token) { try 84 if (token.type === Keyword) { catch 84 console.log(token.value); delete 72 } throw 38 }); switch 35 }); continue 25 default 15instanceof 14 do 12 void 10 finally 4 http://ariya.oﬁlabs.com/2012/03/most-popular-javascript-keywords.html

Most Popular Statements var fs = require(fs),ExpressionStatement 6728 esprima = require(esprima), BlockStatement 6353 files = process.argv.splice(2); IfStatement 3063 ReturnStatement 2878 files.forEach(function (filename) { VariableDeclaration 2116 var content = fs.readFileSync(filename, utf-8), syntax = esprima.parse(content); FunctionDeclaration 371 ForStatement 293 JSON.stringify(syntax, function (key, value) { ForInStatement 143 if (key === type) { WhileStatement 131 if (value.match(/Declaration$/) || BreakStatement 115 value.match(/Statement$/)) { TryStatement 84 console.log(value); EmptyStatement 66 } } ThrowStatement 38 return value; SwitchStatement 35 }); ContinueStatement 25 }); DoWhileStatement 12 LabeledStatement 6 http://ariya.oﬁlabs.com/2012/04/most-popular-javascript-statements.html

Identifier Length Distribution750 mean of the identifier length is 8.27 characters500 prototype-1.7.0.0.js SCRIPT_ELEMENT_REJECTS_TEXTNODE_APPENDING prototype-1.7.0.0.js MOUSEENTER_MOUSELEAVE_EVENTS_SUPPORTED jquery-1.7.1.js subtractsBorderForOverflowNotVisible jquery.mobile-1.0.js getClosestElementWithVirtualBinding250 prototype-1.7.0.0.js HAS_EXTENDED_CREATE_ELEMENT_SYNTAX 0 0 5 10 15 20 25 30 35 40 45 http://ariya.ofilabs.com/2012/05/javascript-identifier-length-distribution.html

“Code Linting” var fs = require(fs), esprima = require(./esprima), files = process.argv.splice(2); Not a strict equal files.forEach(function (filename) { var content = fs.readFileSync(filename, utf-8), syntax = esprima.parse(content, { loc: true });if (x == 9) { // do Something JSON.stringify(syntax, function (key, value) {} if (key === test && value.operator === ==) console.log(Line, value.loc.start.line); return value; }); }); Another example: next-generation JSHint https://github.com/jshint/jshint-next

Strict Mode Checking Duplicate data property in object literal not allowed in strict mode use strict; block = { color: blue, height: 20, width: 10, color: red }; http://ariya.oﬁlabs.com/2012/03/strict-mode-checks-with-esprima.html

http://ariya.oﬁlabs.com/2011/08/hall-of-api-shame-boolean-trap.html“Boolean Trap” Finder var volumeSlider = new Slider(false); Obfuscated choice component.setHidden(false); Double-negative filter.setCaseInsensitive(false);Can you make up your mind? treeItem.setState(true, false); event.initKeyEvent("keypress", true, true, null, null, The more the merrier? false, false, false, false, 9, 0);

Style Formatter https://github.com/fawek/codepainter Sample code Source CodePainter Formatted source Infer coding styles Indentation Quote for string literal Whitespace

https://github.com/Constellation/escodegenRewrite and Regenerate answer = 42; var syntax = esprima.parse(answer = 42;); syntax.body[0].expression.right.value = 1337; escodegen.generate(syntax) answer = 1337;

Code Coverage https://github.com/itay/node-cover https://github.com/coveraje/coveraje x = 42; https://github.com/pmlopes/coberturajs if (false) x = -1; http://ariya.oﬁlabs.com/2012/03/javascript-code-coverage-and-esprima.html

http://itay.github.com/snug_codecoverage_slides/Instrumentation for Coverage { __statement_ZyyqFc(1); var a = 5; Statement var a = 5; } { __statement_ZyyqFc(2); Expression foo(); __expression_kC$jur(3), foo(); } function foo() { function foo() { Block ... __block_n53cJc(1); }; ... }

Non-Destructive Partial Source Modification Do not remove comments Intact Preserve indentation & other formatting Modiﬁed Add “contextual” information Inject or change function invocation

String Literal Quotes May need proper escaping console.log("Hello") console.log(Hello) List of tokens [ { type: "Identifier", value: "console", range: [0, 7] }, { type: "Punctuator", value: ".", range: [7, 8] }, { type: "Identifier", value: "log", range: [8, 11] }, { type: "Punctuator", value: "(", range: [11, 12] }, { type: "String", value: ""Hello"", range: [12, 19] }, { type: "Punctuator", value: ")", range: [19, 19] } ] http://ariya.oﬁlabs.com/2012/02/from-double-quotes-to-single-quotes.html

Tracking the Scalability Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k; } Array.prototype.swap = function (i, j) { Log({ name: Array.prototype.swap, lineNumber: 1, range: [23, 94] }); var k = this[i]; this[i] = this[j]; this[j] = k; } http://ariya.oﬁlabs.com/2012/01/scalable-web-apps-the-complexity-issue.html

Execution Tracing 4640 function calls jQuery Mobile startup log https://gist.github.com/1823129 jquery.js 26 jQuery jquery.js 103 init undefined, undefined, [object Object] jquery.js 274 each (Function) jquery.js 631 each [object Object], (Function), undefined jquery.js 495 isFunction [object Object] jquery.js 512 type [object Object] jquery.mobile.js 1857 [Anonymous] jquery.mobile.js 642 [Anonymous] jquery.mobile.js 624 enableMouseBindings jquery.mobile.js 620 disableTouchBindings http://ariya.oﬁlabs.com/2012/02/tracking-javascript-execution-during-startup.html

http://esprima.googlecode.com/git-history/harmony/demo/transpile.htmlTranspilation // Object literal property shorthand. function Point(x, y) { Harmony return { x, y }; } Intact // Object literal property shorthand. ES 5.1 function Point(x, y) { return { x: x, y: y }; } Inserted fragment

Application Structure Class manifestExt.define(My.sample.Person, { name: Mr. Unknown, age: 42, { className: My.sample.Person, constructor: function(name) {}, functions: [walk, run], properties: [name, age] walk: function(steps) {} } run: function(steps) {}}); http://www.sencha.com/learn/sencha-class-system/

Code Outline EclipseFunctions Variables

Content Assist (aka Autocomplete, aka “Intellisense”) Eclipse http://contraptionsforprogramming.blogspot.com/2012/02/better-javascript-content-assist-in.html

Copy Paste Mistakefunction inside(point, rect) { return (point.x >= rect.x1) && (point.y >= rect.y1) && (point.x <= rect.x2) && (point.y <= rect.y1);} Wrong check

Refactoring Helper // Add two numbers // Add two numbers function add(firt, two) { function add(first, two) { return firt + two; return first + two; } }

Future Tree traversal Scope analysis “Richer” syntax treeSyntax transformation Pattern Matching

Conclusion

Smart editing Source transformation Instrumentation Modern Miniﬁcation & obfuscation ParserCode coverage Documentation generator Conditional contracts Dependency analysis

Thank You! ariya.hidayat@gmail.com ariya.oﬁlabs.com @AriyaHidayat

https://twitter.com	75
https://si0.twimg.com	4
https://www.rebelmouse.com	1

JavaScript Parser Infrastructure for Code Quality Analysis

by Ariya Hidayat on May 31, 2012

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 80

Statistics

JavaScript Parser Infrastructure for Code Quality Analysis Presentation Transcript