JavaScript Parser  Infrastructure forCode Quality Analysis         Ariya Hidayat     Twitter: @AriyaHidayat     http://ari...
Who Am I
Do you...    have a coding style/guidelines?    use code quality tool such as JSLint or JSHint?    actively prevent per...
Code Quality
High Quality: Practical Aspects                           Do not provoke ambiguities   Avoid silly mistakes               ...
Better Use of the Tools                                             Not everyone is                                       ...
From Spelling Checker to Grammar Enforcement Agent               Your so wrong,             therefore you loose!          ...
Adaptive Quality Criteria         Explicit                     ImplicitCustomize analysis options   Infer from high-qualit...
Next-Generation Code Quality Tools   To boldly analyze what no man has analyzed before...
Parser Infrastructure
JavaScript in the Browser                                  User Interface                                                 ...
JavaScript Engine Building Blocks                                                           Built-in objects,             ...
Tokenization                         identifier           number           var answer = 42;               keyword          ...
Syntax Tree                    Variable Declaration              Identifier              Literal Constant               ans...
JavaScript Parser Written in JavaScript             Esprima            UglifyJS                             Narcissus     ...
Produce Correct Output                          ECMA-262 compliant       use strict;      Automatic semicolon insertion ...
Heavily Tested                  > 500 unit tests                  Compatibility tests                  100% code covera...
Sensible Syntax Tree                      https://developer.mozilla.org/en/SpiderMonkey/Parser_API                        ...
Syntax Tree Visualization     answer = 42
Specification, Parser Code, Syntax Tree               ECMA-262 Annex A.4        while ( Expression ) Statement  function p...
High Performance                                   http://esprima.org/test/compare.html                                   ...
Syntax Node Location   {       type: "ExpressionStatement",       expression: {           type: "AssignmentExpression",   ...
Fit for Code Regeneration                https://github.com/Constellation/escodegen Source     Esprima                    ...
Easily Tweaked                LLJS: Low Level JavaScript   http://mbebenita.github.com/LLJS/                 Block scope  ...
Error Tolerant                                             Useful for IDE, editors, ...              Mismatched quote     ...
Handle the Comments                           // Life, Universe, and Everything                           answer = 42     ...
Forward Looking                               Experimental ‘harmony’ branch  Object initializer shorthand    Destructuring...
Code Quality Tools
Syntax Tree Visualization                      Block statement http://esprima.org/demo/parse.html
Most Popular Keywords        this                                           3229  function                                ...
Most Popular Statements                                                                           var fs = require(fs),Exp...
Identifier Length Distribution750                                                      mean of the identifier length is 8.2...
“Code Linting”                                          var fs = require(fs),                                             ...
Strict Mode Checking                                             Duplicate data property in object                        ...
http://ariya.ofilabs.com/2011/08/hall-of-api-shame-boolean-trap.html“Boolean Trap” Finder                              var ...
Style Formatter                                     https://github.com/fawek/codepainter                           Sample ...
https://github.com/Constellation/escodegenRewrite and Regenerate                          answer = 42;            var synt...
Code Coverage                                             https://github.com/itay/node-cover                              ...
http://itay.github.com/snug_codecoverage_slides/Instrumentation for Coverage                                      {       ...
Non-Destructive Partial Source Modification                                    Do not remove comments                     ...
String Literal Quotes                             May need proper escaping            console.log("Hello")                ...
Tracking the Scalability            Array.prototype.swap = function (i, j) {                var k = this[i]; this[i] = thi...
Execution Tracing                                                 4640 function calls                      jQuery Mobile s...
http://esprima.googlecode.com/git-history/harmony/demo/transpile.htmlTranspilation                // Object literal proper...
Application Structure                                                      Class manifestExt.define(My.sample.Person, {   ...
Code Outline   EclipseFunctions   Variables
Content Assist (aka Autocomplete, aka “Intellisense”)                                                       Eclipse       ...
Copy Paste Mistakefunction inside(point, rect) {    return (point.x >= rect.x1) && (point.y >= rect.y1) &&           (poin...
Refactoring Helper    // Add two numbers          // Add two numbers    function add(firt, two) {   function add(first, tw...
Future                        Tree traversal    Scope analysis                         “Richer” syntax treeSyntax transfor...
Conclusion
Smart editing                                          Source transformation   Instrumentation                            ...
Thank You!             ariya.hidayat@gmail.com             ariya.ofilabs.com             @AriyaHidayat
Upcoming SlideShare
Loading in …5
×

JavaScript Parser Infrastructure for Code Quality Analysis

8,278 views

Published on

Published in: Technology
2 Comments
26 Likes
Statistics
Notes
No Downloads
Views
Total views
8,278
On SlideShare
0
From Embeds
0
Number of Embeds
134
Actions
Shares
0
Downloads
72
Comments
2
Likes
26
Embeds 0
No embeds

No notes for slide

JavaScript Parser Infrastructure for Code Quality Analysis

  1. 1. JavaScript Parser Infrastructure forCode Quality Analysis Ariya Hidayat Twitter: @AriyaHidayat http://ariya.ofilabs.com
  2. 2. Who Am I
  3. 3. Do you...  have a coding style/guidelines?  use code quality tool such as JSLint or JSHint?  actively prevent performance & coverage regressions?  wish you could write your own quality rules?
  4. 4. Code Quality
  5. 5. High Quality: Practical Aspects Do not provoke ambiguities Avoid silly mistakes Learn better code pattern Write readable code Improve future maintenance
  6. 6. Better Use of the Tools Not everyone is a JavaScript ninja Engineer Tools •Boring •Repetitive Feedback Cycle •Time-consuming
  7. 7. From Spelling Checker to Grammar Enforcement Agent Your so wrong, therefore you loose! No misspelled word. Wrong choice of words!
  8. 8. Adaptive Quality Criteria Explicit ImplicitCustomize analysis options Infer from high-quality sampleDefine new sets of rules Observe the engineer’s behavior
  9. 9. Next-Generation Code Quality Tools To boldly analyze what no man has analyzed before...
  10. 10. Parser Infrastructure
  11. 11. JavaScript in the Browser User Interface Data Persistence Browser Engine Render Engine JavaScript Graphics Networking I/O Engine Stack
  12. 12. JavaScript Engine Building Blocks Built-in objects, Runtime host objects, ... Syntax Tree Virtual Source Parser Machine/ Interpreter Fast and conservative
  13. 13. Tokenization identifier number var answer = 42; keyword equal sign semicolon
  14. 14. Syntax Tree Variable Declaration Identifier Literal Constant answer 42
  15. 15. JavaScript Parser Written in JavaScript Esprima UglifyJS Narcissus ZeParser Traceur Es-Lab
  16. 16. Produce Correct Output  ECMA-262 compliant use strict;  Automatic semicolon insertion var 日本語 = 1  Strict Mode, e.g. “use strict” return 日本語  Unicode for identifiers
  17. 17. Heavily Tested  > 500 unit tests  Compatibility tests  100% code coverage  Performance tests Enforced during development
  18. 18. Sensible Syntax Tree https://developer.mozilla.org/en/SpiderMonkey/Parser_API { type: "Program", body: [ { type: "ExpressionStatement", expression: { answer = 42 type: "AssignmentExpression", operator: "=", left: { type: "Identifier", name: "answer" }, right: { type: "Literal", value: 42 } http://esprima.org/demo/parse.html } } ] Try online! }
  19. 19. Syntax Tree Visualization answer = 42
  20. 20. Specification, Parser Code, Syntax Tree ECMA-262 Annex A.4 while ( Expression ) Statement function parseWhileStatement() { var test, body;   expectKeyword(while); expect((); test = parseExpression(); expect()); body = parseStatement();   return { type: Syntax.WhileStatement, test: test, body: body }; }
  21. 21. High Performance http://esprima.org/test/compare.html Speed Comparison Chrome 18 Internet Explorer 9 Esprima 233 ms 567 ms UglifyJS parse-js 620 ms 922 ms Benchmark corpus: jQuery, Prototype, MooTools, ExtCore, ...
  22. 22. Syntax Node Location { type: "ExpressionStatement", expression: { type: "AssignmentExpression", operator: "=", left: { answer = 42 type: "Identifier", name: "answer", range: [0, 6] }, right: { type: "Literal", value: 42, range: [9, 11] }, range: [0, 11] }, range: [0, 11] }
  23. 23. Fit for Code Regeneration https://github.com/Constellation/escodegen Source Esprima Escodegen Source Syntax Tree Shorten variable name Syntax Transformation Automatically inline short function Obfuscate
  24. 24. Easily Tweaked LLJS: Low Level JavaScript http://mbebenita.github.com/LLJS/ Block scope let x = 0; let u8 flag; Data types let i32 position; struct Point { int x, y; }; Pointers let u16 *p = q;
  25. 25. Error Tolerant Useful for IDE, editors, ... Mismatched quote var msg = "Hello’; Too many dots person..age = 18;   Incomplete, still typing if (person.   Strict mode violation use strict; with (person) { }
  26. 26. Handle the Comments // Life, Universe, and Everything answer = 42 comments: [ { range: [0, 34], type: "Line", value: " Life, Universe, and Everything" } ] Documentation tool Code annotation https://github.com/thejohnfreeman/jfdoc https://github.com/goatslacker/node-typedjs
  27. 27. Forward Looking Experimental ‘harmony’ branch Object initializer shorthand Destructuring assignment var point = {x, y}; point = {14, 3, 77}; {x, y, z} = point; Module Block scope module MathExtra { // do something { let x; } const y = 0; }
  28. 28. Code Quality Tools
  29. 29. Syntax Tree Visualization Block statement http://esprima.org/demo/parse.html
  30. 30. Most Popular Keywords this 3229 function 3108 if 3063 return 2878 var 2116 var fs = require(fs), else 562 esprima = require(esprima), for 436 files = process.argv.splice(2); new 232   in 225 files.forEach(function (filename) { typeof 188 var content = fs.readFileSync(filename, utf-8), while 143 tokens = esprima.parse(content, { tokens: true }).tokens; case 122   break 115 tokens.forEach(function (token) { try 84 if (token.type === Keyword) { catch 84 console.log(token.value); delete 72 } throw 38 }); switch 35 }); continue 25 default 15instanceof 14 do 12 void 10 finally 4 http://ariya.ofilabs.com/2012/03/most-popular-javascript-keywords.html
  31. 31. Most Popular Statements var fs = require(fs),ExpressionStatement 6728 esprima = require(esprima), BlockStatement 6353 files = process.argv.splice(2); IfStatement 3063   ReturnStatement 2878 files.forEach(function (filename) { VariableDeclaration 2116 var content = fs.readFileSync(filename, utf-8), syntax = esprima.parse(content); FunctionDeclaration 371   ForStatement 293 JSON.stringify(syntax, function (key, value) { ForInStatement 143 if (key === type) { WhileStatement 131 if (value.match(/Declaration$/) || BreakStatement 115 value.match(/Statement$/)) { TryStatement 84 console.log(value); EmptyStatement 66 } } ThrowStatement 38 return value; SwitchStatement 35 }); ContinueStatement 25 }); DoWhileStatement 12 LabeledStatement 6 http://ariya.ofilabs.com/2012/04/most-popular-javascript-statements.html
  32. 32. Identifier Length Distribution750 mean of the identifier length is 8.27 characters500 prototype-1.7.0.0.js SCRIPT_ELEMENT_REJECTS_TEXTNODE_APPENDING prototype-1.7.0.0.js MOUSEENTER_MOUSELEAVE_EVENTS_SUPPORTED jquery-1.7.1.js subtractsBorderForOverflowNotVisible jquery.mobile-1.0.js getClosestElementWithVirtualBinding250 prototype-1.7.0.0.js HAS_EXTENDED_CREATE_ELEMENT_SYNTAX 0 0 5 10 15 20 25 30 35 40 45 http://ariya.ofilabs.com/2012/05/javascript-identifier-length-distribution.html
  33. 33. “Code Linting” var fs = require(fs), esprima = require(./esprima), files = process.argv.splice(2);   Not a strict equal files.forEach(function (filename) { var content = fs.readFileSync(filename, utf-8), syntax = esprima.parse(content, { loc: true });if (x == 9) {   // do Something JSON.stringify(syntax, function (key, value) {} if (key === test && value.operator === ==) console.log(Line, value.loc.start.line); return value; }); }); Another example: next-generation JSHint https://github.com/jshint/jshint-next
  34. 34. Strict Mode Checking Duplicate data property in object literal not allowed in strict mode use strict; block = { color: blue, height: 20, width: 10, color: red }; http://ariya.ofilabs.com/2012/03/strict-mode-checks-with-esprima.html
  35. 35. http://ariya.ofilabs.com/2011/08/hall-of-api-shame-boolean-trap.html“Boolean Trap” Finder var volumeSlider = new Slider(false); Obfuscated choice component.setHidden(false); Double-negative filter.setCaseInsensitive(false);Can you make up your mind? treeItem.setState(true, false); event.initKeyEvent("keypress", true, true, null, null, The more the merrier? false, false, false, false, 9, 0);
  36. 36. Style Formatter https://github.com/fawek/codepainter Sample code Source CodePainter Formatted source Infer coding styles Indentation Quote for string literal Whitespace
  37. 37. https://github.com/Constellation/escodegenRewrite and Regenerate answer = 42; var syntax = esprima.parse(answer = 42;); syntax.body[0].expression.right.value = 1337; escodegen.generate(syntax) answer = 1337;
  38. 38. Code Coverage https://github.com/itay/node-cover https://github.com/coveraje/coveraje x = 42; https://github.com/pmlopes/coberturajs if (false) x = -1; http://ariya.ofilabs.com/2012/03/javascript-code-coverage-and-esprima.html
  39. 39. http://itay.github.com/snug_codecoverage_slides/Instrumentation for Coverage { __statement_ZyyqFc(1); var a = 5; Statement var a = 5; } { __statement_ZyyqFc(2); Expression foo(); __expression_kC$jur(3), foo(); } function foo() { function foo() { Block ... __block_n53cJc(1); }; ... }
  40. 40. Non-Destructive Partial Source Modification Do not remove comments Intact Preserve indentation & other formatting Modified Add “contextual” information Inject or change function invocation
  41. 41. String Literal Quotes May need proper escaping console.log("Hello") console.log(Hello) List of tokens [ { type: "Identifier", value: "console", range: [0, 7] }, { type: "Punctuator", value: ".", range: [7, 8] }, { type: "Identifier", value: "log", range: [8, 11] }, { type: "Punctuator", value: "(", range: [11, 12] }, { type: "String", value: ""Hello"", range: [12, 19] }, { type: "Punctuator", value: ")", range: [19, 19] } ] http://ariya.ofilabs.com/2012/02/from-double-quotes-to-single-quotes.html
  42. 42. Tracking the Scalability Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k; } Array.prototype.swap = function (i, j) { Log({ name: Array.prototype.swap, lineNumber: 1, range: [23, 94] }); var k = this[i]; this[i] = this[j]; this[j] = k; } http://ariya.ofilabs.com/2012/01/scalable-web-apps-the-complexity-issue.html
  43. 43. Execution Tracing 4640 function calls jQuery Mobile startup log https://gist.github.com/1823129 jquery.js 26 jQuery jquery.js 103 init undefined, undefined, [object Object] jquery.js 274 each (Function) jquery.js 631 each [object Object], (Function), undefined jquery.js 495 isFunction [object Object] jquery.js 512 type [object Object] jquery.mobile.js 1857 [Anonymous] jquery.mobile.js 642 [Anonymous] jquery.mobile.js 624 enableMouseBindings jquery.mobile.js 620 disableTouchBindings http://ariya.ofilabs.com/2012/02/tracking-javascript-execution-during-startup.html
  44. 44. http://esprima.googlecode.com/git-history/harmony/demo/transpile.htmlTranspilation // Object literal property shorthand. function Point(x, y) { Harmony return { x, y }; } Intact // Object literal property shorthand. ES 5.1 function Point(x, y) { return { x: x, y: y }; } Inserted fragment
  45. 45. Application Structure Class manifestExt.define(My.sample.Person, { name: Mr. Unknown,  age: 42, { className: My.sample.Person, constructor: function(name) {}, functions: [walk, run],  properties: [name, age] walk: function(steps) {} } run: function(steps) {}}); http://www.sencha.com/learn/sencha-class-system/
  46. 46. Code Outline EclipseFunctions Variables
  47. 47. Content Assist (aka Autocomplete, aka “Intellisense”) Eclipse http://contraptionsforprogramming.blogspot.com/2012/02/better-javascript-content-assist-in.html
  48. 48. Copy Paste Mistakefunction inside(point, rect) { return (point.x >= rect.x1) && (point.y >= rect.y1) && (point.x <= rect.x2) && (point.y <= rect.y1);} Wrong check
  49. 49. Refactoring Helper // Add two numbers // Add two numbers function add(firt, two) { function add(first, two) { return firt + two; return first + two; } }
  50. 50. Future Tree traversal Scope analysis “Richer” syntax treeSyntax transformation Pattern Matching
  51. 51. Conclusion
  52. 52. Smart editing Source transformation Instrumentation Modern Minification & obfuscation ParserCode coverage Documentation generator Conditional contracts Dependency analysis
  53. 53. Thank You! ariya.hidayat@gmail.com ariya.ofilabs.com @AriyaHidayat

×