JavaScript Parser Infrastructure for Code Quality Analysis
Upcoming SlideShare
Loading in...5
×
 

JavaScript Parser Infrastructure for Code Quality Analysis

on

  • 6,181 views

 

Statistics

Views

Total Views
6,181
Views on SlideShare
5,929
Embed Views
252

Actions

Likes
22
Downloads
48
Comments
2

4 Embeds 252

https://twitter.com 246
https://si0.twimg.com 4
https://www.rebelmouse.com 1
https://tweetdeck.twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • PMD is written in Java and I'm fairly sure its ECMAScript support is not ES.Next/Harmony-ready.
    Are you sure you want to
    Your message goes here
    Processing…
  • Nice presentation. Did you ever looked at PMD ?http://pmd.sourceforge.net/. it does provide a javascript ast parser, and coding rules you can extend through xpath...
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

JavaScript Parser Infrastructure for Code Quality Analysis JavaScript Parser Infrastructure for Code Quality Analysis Presentation Transcript

  • JavaScript Parser Infrastructure forCode Quality Analysis Ariya Hidayat Twitter: @AriyaHidayat http://ariya.ofilabs.com
  • Who Am I
  • Do you...  have a coding style/guidelines?  use code quality tool such as JSLint or JSHint?  actively prevent performance & coverage regressions?  wish you could write your own quality rules?
  • Code Quality
  • High Quality: Practical Aspects Do not provoke ambiguities Avoid silly mistakes Learn better code pattern Write readable code Improve future maintenance
  • Better Use of the Tools Not everyone is a JavaScript ninja Engineer Tools •Boring •Repetitive Feedback Cycle •Time-consuming
  • From Spelling Checker to Grammar Enforcement Agent Your so wrong, therefore you loose! No misspelled word. Wrong choice of words!
  • Adaptive Quality Criteria Explicit ImplicitCustomize analysis options Infer from high-quality sampleDefine new sets of rules Observe the engineer’s behavior
  • Next-Generation Code Quality Tools To boldly analyze what no man has analyzed before...
  • Parser Infrastructure
  • JavaScript in the Browser User Interface Data Persistence Browser Engine Render Engine JavaScript Graphics Networking I/O Engine Stack
  • JavaScript Engine Building Blocks Built-in objects, Runtime host objects, ... Syntax Tree Virtual Source Parser Machine/ Interpreter Fast and conservative
  • Tokenization identifier number var answer = 42; keyword equal sign semicolon
  • Syntax Tree Variable Declaration Identifier Literal Constant answer 42
  • JavaScript Parser Written in JavaScript Esprima UglifyJS Narcissus ZeParser Traceur Es-Lab
  • Produce Correct Output  ECMA-262 compliant use strict;  Automatic semicolon insertion var 日本語 = 1  Strict Mode, e.g. “use strict” return 日本語  Unicode for identifiers
  • Heavily Tested  > 500 unit tests  Compatibility tests  100% code coverage  Performance tests Enforced during development
  • Sensible Syntax Tree https://developer.mozilla.org/en/SpiderMonkey/Parser_API { type: "Program", body: [ { type: "ExpressionStatement", expression: { answer = 42 type: "AssignmentExpression", operator: "=", left: { type: "Identifier", name: "answer" }, right: { type: "Literal", value: 42 } http://esprima.org/demo/parse.html } } ] Try online! }
  • Syntax Tree Visualization answer = 42
  • Specification, Parser Code, Syntax Tree ECMA-262 Annex A.4 while ( Expression ) Statement function parseWhileStatement() { var test, body;   expectKeyword(while); expect((); test = parseExpression(); expect()); body = parseStatement();   return { type: Syntax.WhileStatement, test: test, body: body }; }
  • High Performance http://esprima.org/test/compare.html Speed Comparison Chrome 18 Internet Explorer 9 Esprima 233 ms 567 ms UglifyJS parse-js 620 ms 922 ms Benchmark corpus: jQuery, Prototype, MooTools, ExtCore, ...
  • Syntax Node Location { type: "ExpressionStatement", expression: { type: "AssignmentExpression", operator: "=", left: { answer = 42 type: "Identifier", name: "answer", range: [0, 6] }, right: { type: "Literal", value: 42, range: [9, 11] }, range: [0, 11] }, range: [0, 11] }
  • Fit for Code Regeneration https://github.com/Constellation/escodegen Source Esprima Escodegen Source Syntax Tree Shorten variable name Syntax Transformation Automatically inline short function Obfuscate
  • Easily Tweaked LLJS: Low Level JavaScript http://mbebenita.github.com/LLJS/ Block scope let x = 0; let u8 flag; Data types let i32 position; struct Point { int x, y; }; Pointers let u16 *p = q;
  • Error Tolerant Useful for IDE, editors, ... Mismatched quote var msg = "Hello’; Too many dots person..age = 18;   Incomplete, still typing if (person.   Strict mode violation use strict; with (person) { }
  • Handle the Comments // Life, Universe, and Everything answer = 42 comments: [ { range: [0, 34], type: "Line", value: " Life, Universe, and Everything" } ] Documentation tool Code annotation https://github.com/thejohnfreeman/jfdoc https://github.com/goatslacker/node-typedjs
  • Forward Looking Experimental ‘harmony’ branch Object initializer shorthand Destructuring assignment var point = {x, y}; point = {14, 3, 77}; {x, y, z} = point; Module Block scope module MathExtra { // do something { let x; } const y = 0; }
  • Code Quality Tools
  • Syntax Tree Visualization Block statement http://esprima.org/demo/parse.html
  • Most Popular Keywords this 3229 function 3108 if 3063 return 2878 var 2116 var fs = require(fs), else 562 esprima = require(esprima), for 436 files = process.argv.splice(2); new 232   in 225 files.forEach(function (filename) { typeof 188 var content = fs.readFileSync(filename, utf-8), while 143 tokens = esprima.parse(content, { tokens: true }).tokens; case 122   break 115 tokens.forEach(function (token) { try 84 if (token.type === Keyword) { catch 84 console.log(token.value); delete 72 } throw 38 }); switch 35 }); continue 25 default 15instanceof 14 do 12 void 10 finally 4 http://ariya.ofilabs.com/2012/03/most-popular-javascript-keywords.html
  • Most Popular Statements var fs = require(fs),ExpressionStatement 6728 esprima = require(esprima), BlockStatement 6353 files = process.argv.splice(2); IfStatement 3063   ReturnStatement 2878 files.forEach(function (filename) { VariableDeclaration 2116 var content = fs.readFileSync(filename, utf-8), syntax = esprima.parse(content); FunctionDeclaration 371   ForStatement 293 JSON.stringify(syntax, function (key, value) { ForInStatement 143 if (key === type) { WhileStatement 131 if (value.match(/Declaration$/) || BreakStatement 115 value.match(/Statement$/)) { TryStatement 84 console.log(value); EmptyStatement 66 } } ThrowStatement 38 return value; SwitchStatement 35 }); ContinueStatement 25 }); DoWhileStatement 12 LabeledStatement 6 http://ariya.ofilabs.com/2012/04/most-popular-javascript-statements.html
  • Identifier Length Distribution750 mean of the identifier length is 8.27 characters500 prototype-1.7.0.0.js SCRIPT_ELEMENT_REJECTS_TEXTNODE_APPENDING prototype-1.7.0.0.js MOUSEENTER_MOUSELEAVE_EVENTS_SUPPORTED jquery-1.7.1.js subtractsBorderForOverflowNotVisible jquery.mobile-1.0.js getClosestElementWithVirtualBinding250 prototype-1.7.0.0.js HAS_EXTENDED_CREATE_ELEMENT_SYNTAX 0 0 5 10 15 20 25 30 35 40 45 http://ariya.ofilabs.com/2012/05/javascript-identifier-length-distribution.html
  • “Code Linting” var fs = require(fs), esprima = require(./esprima), files = process.argv.splice(2);   Not a strict equal files.forEach(function (filename) { var content = fs.readFileSync(filename, utf-8), syntax = esprima.parse(content, { loc: true });if (x == 9) {   // do Something JSON.stringify(syntax, function (key, value) {} if (key === test && value.operator === ==) console.log(Line, value.loc.start.line); return value; }); }); Another example: next-generation JSHint https://github.com/jshint/jshint-next
  • Strict Mode Checking Duplicate data property in object literal not allowed in strict mode use strict; block = { color: blue, height: 20, width: 10, color: red }; http://ariya.ofilabs.com/2012/03/strict-mode-checks-with-esprima.html
  • http://ariya.ofilabs.com/2011/08/hall-of-api-shame-boolean-trap.html“Boolean Trap” Finder var volumeSlider = new Slider(false); Obfuscated choice component.setHidden(false); Double-negative filter.setCaseInsensitive(false);Can you make up your mind? treeItem.setState(true, false); event.initKeyEvent("keypress", true, true, null, null, The more the merrier? false, false, false, false, 9, 0);
  • Style Formatter https://github.com/fawek/codepainter Sample code Source CodePainter Formatted source Infer coding styles Indentation Quote for string literal Whitespace
  • https://github.com/Constellation/escodegenRewrite and Regenerate answer = 42; var syntax = esprima.parse(answer = 42;); syntax.body[0].expression.right.value = 1337; escodegen.generate(syntax) answer = 1337;
  • Code Coverage https://github.com/itay/node-cover https://github.com/coveraje/coveraje x = 42; https://github.com/pmlopes/coberturajs if (false) x = -1; http://ariya.ofilabs.com/2012/03/javascript-code-coverage-and-esprima.html
  • http://itay.github.com/snug_codecoverage_slides/Instrumentation for Coverage { __statement_ZyyqFc(1); var a = 5; Statement var a = 5; } { __statement_ZyyqFc(2); Expression foo(); __expression_kC$jur(3), foo(); } function foo() { function foo() { Block ... __block_n53cJc(1); }; ... }
  • Non-Destructive Partial Source Modification Do not remove comments Intact Preserve indentation & other formatting Modified Add “contextual” information Inject or change function invocation
  • String Literal Quotes May need proper escaping console.log("Hello") console.log(Hello) List of tokens [ { type: "Identifier", value: "console", range: [0, 7] }, { type: "Punctuator", value: ".", range: [7, 8] }, { type: "Identifier", value: "log", range: [8, 11] }, { type: "Punctuator", value: "(", range: [11, 12] }, { type: "String", value: ""Hello"", range: [12, 19] }, { type: "Punctuator", value: ")", range: [19, 19] } ] http://ariya.ofilabs.com/2012/02/from-double-quotes-to-single-quotes.html
  • Tracking the Scalability Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k; } Array.prototype.swap = function (i, j) { Log({ name: Array.prototype.swap, lineNumber: 1, range: [23, 94] }); var k = this[i]; this[i] = this[j]; this[j] = k; } http://ariya.ofilabs.com/2012/01/scalable-web-apps-the-complexity-issue.html
  • Execution Tracing 4640 function calls jQuery Mobile startup log https://gist.github.com/1823129 jquery.js 26 jQuery jquery.js 103 init undefined, undefined, [object Object] jquery.js 274 each (Function) jquery.js 631 each [object Object], (Function), undefined jquery.js 495 isFunction [object Object] jquery.js 512 type [object Object] jquery.mobile.js 1857 [Anonymous] jquery.mobile.js 642 [Anonymous] jquery.mobile.js 624 enableMouseBindings jquery.mobile.js 620 disableTouchBindings http://ariya.ofilabs.com/2012/02/tracking-javascript-execution-during-startup.html
  • http://esprima.googlecode.com/git-history/harmony/demo/transpile.htmlTranspilation // Object literal property shorthand. function Point(x, y) { Harmony return { x, y }; } Intact // Object literal property shorthand. ES 5.1 function Point(x, y) { return { x: x, y: y }; } Inserted fragment
  • Application Structure Class manifestExt.define(My.sample.Person, { name: Mr. Unknown,  age: 42, { className: My.sample.Person, constructor: function(name) {}, functions: [walk, run],  properties: [name, age] walk: function(steps) {} } run: function(steps) {}}); http://www.sencha.com/learn/sencha-class-system/
  • Code Outline EclipseFunctions Variables
  • Content Assist (aka Autocomplete, aka “Intellisense”) Eclipse http://contraptionsforprogramming.blogspot.com/2012/02/better-javascript-content-assist-in.html
  • Copy Paste Mistakefunction inside(point, rect) { return (point.x >= rect.x1) && (point.y >= rect.y1) && (point.x <= rect.x2) && (point.y <= rect.y1);} Wrong check
  • Refactoring Helper // Add two numbers // Add two numbers function add(firt, two) { function add(first, two) { return firt + two; return first + two; } }
  • Future Tree traversal Scope analysis “Richer” syntax treeSyntax transformation Pattern Matching
  • Conclusion
  • Smart editing Source transformation Instrumentation Modern Minification & obfuscation ParserCode coverage Documentation generator Conditional contracts Dependency analysis
  • Thank You! ariya.hidayat@gmail.com ariya.ofilabs.com @AriyaHidayat