Advertisement
Advertisement

More Related Content

Advertisement
Advertisement

JavaScript Parser Infrastructure for Code Quality Analysis

  1. JavaScript Parser Infrastructure for Code Quality Analysis Ariya Hidayat Twitter: @AriyaHidayat http://ariya.ofilabs.com
  2. Who Am I
  3. Do you...  have a coding style/guidelines?  use code quality tool such as JSLint or JSHint?  actively prevent performance & coverage regressions?  wish you could write your own quality rules?
  4. Code Quality
  5. High Quality: Practical Aspects Do not provoke ambiguities Avoid silly mistakes Learn better code pattern Write readable code Improve future maintenance
  6. Better Use of the Tools Not everyone is a JavaScript ninja Engineer Tools •Boring •Repetitive Feedback Cycle •Time-consuming
  7. From Spelling Checker to Grammar Enforcement Agent Your so wrong, therefore you loose! No misspelled word. Wrong choice of words!
  8. Adaptive Quality Criteria Explicit Implicit Customize analysis options Infer from high-quality sample Define new sets of rules Observe the engineer’s behavior
  9. Next-Generation Code Quality Tools To boldly analyze what no man has analyzed before...
  10. Parser Infrastructure
  11. JavaScript in the Browser User Interface Data Persistence Browser Engine Render Engine JavaScript Graphics Networking I/O Engine Stack
  12. JavaScript Engine Building Blocks Built-in objects, Runtime host objects, ... Syntax Tree Virtual Source Parser Machine/ Interpreter Fast and conservative
  13. Tokenization identifier number var answer = 42; keyword equal sign semicolon
  14. Syntax Tree Variable Declaration Identifier Literal Constant answer 42
  15. JavaScript Parser Written in JavaScript Esprima UglifyJS Narcissus ZeParser Traceur Es-Lab
  16. Produce Correct Output  ECMA-262 compliant 'use strict';  Automatic semicolon insertion var 日本語 = 1  Strict Mode, e.g. “use strict” return 日本語  Unicode for identifiers
  17. Heavily Tested  > 500 unit tests  Compatibility tests  100% code coverage  Performance tests Enforced during development
  18. Sensible Syntax Tree https://developer.mozilla.org/en/SpiderMonkey/Parser_API { type: "Program", body: [ { type: "ExpressionStatement", expression: { answer = 42 type: "AssignmentExpression", operator: "=", left: { type: "Identifier", name: "answer" }, right: { type: "Literal", value: 42 } http://esprima.org/demo/parse.html } } ] Try online! }
  19. Syntax Tree Visualization answer = 42
  20. Specification, Parser Code, Syntax Tree ECMA-262 Annex A.4 while ( Expression ) Statement function parseWhileStatement() { var test, body;   expectKeyword('while'); expect('('); test = parseExpression(); expect(')'); body = parseStatement();   return { type: Syntax.WhileStatement, test: test, body: body }; }
  21. High Performance http://esprima.org/test/compare.html Speed Comparison Chrome 18 Internet Explorer 9 Esprima 233 ms 567 ms UglifyJS parse-js 620 ms 922 ms Benchmark corpus: jQuery, Prototype, MooTools, ExtCore, ...
  22. Syntax Node Location { type: "ExpressionStatement", expression: { type: "AssignmentExpression", operator: "=", left: { answer = 42 type: "Identifier", name: "answer", range: [0, 6] }, right: { type: "Literal", value: 42, range: [9, 11] }, range: [0, 11] }, range: [0, 11] }
  23. Fit for Code Regeneration https://github.com/Constellation/escodegen Source Esprima Escodegen Source Syntax Tree Shorten variable name Syntax Transformation Automatically inline short function Obfuscate
  24. Easily Tweaked LLJS: Low Level JavaScript http://mbebenita.github.com/LLJS/ Block scope let x = 0; let u8 flag; Data types let i32 position; struct Point { int x, y; }; Pointers let u16 *p = q;
  25. Error Tolerant Useful for IDE, editors, ... Mismatched quote var msg = "Hello’; Too many dots person..age = 18;   Incomplete, still typing if (person.   Strict mode violation 'use strict'; with (person) { }
  26. Handle the Comments // Life, Universe, and Everything answer = 42 comments: [ { range: [0, 34], type: "Line", value: " Life, Universe, and Everything" } ] Documentation tool Code annotation https://github.com/thejohnfreeman/jfdoc https://github.com/goatslacker/node-typedjs
  27. Forward Looking Experimental ‘harmony’ branch Object initializer shorthand Destructuring assignment var point = {x, y}; point = {14, 3, 77}; {x, y, z} = point; Module Block scope module MathExtra { // do something { let x; } const y = 0; }
  28. Code Quality Tools
  29. Syntax Tree Visualization Block statement http://esprima.org/demo/parse.html
  30. Most Popular Keywords this 3229 function 3108 if 3063 return 2878 var 2116 var fs = require('fs'), else 562 esprima = require('esprima'), for 436 files = process.argv.splice(2); new 232   in 225 files.forEach(function (filename) { typeof 188 var content = fs.readFileSync(filename, 'utf-8'), while 143 tokens = esprima.parse(content, { tokens: true }).tokens; case 122   break 115 tokens.forEach(function (token) { try 84 if (token.type === 'Keyword') { catch 84 console.log(token.value); delete 72 } throw 38 }); switch 35 }); continue 25 default 15 instanceof 14 do 12 void 10 finally 4 http://ariya.ofilabs.com/2012/03/most-popular-javascript-keywords.html
  31. Most Popular Statements var fs = require('fs'), ExpressionStatement 6728 esprima = require('esprima'), BlockStatement 6353 files = process.argv.splice(2); IfStatement 3063   ReturnStatement 2878 files.forEach(function (filename) { VariableDeclaration 2116 var content = fs.readFileSync(filename, 'utf-8'), syntax = esprima.parse(content); FunctionDeclaration 371   ForStatement 293 JSON.stringify(syntax, function (key, value) { ForInStatement 143 if (key === 'type') { WhileStatement 131 if (value.match(/Declaration$/) || BreakStatement 115 value.match(/Statement$/)) { TryStatement 84 console.log(value); EmptyStatement 66 } } ThrowStatement 38 return value; SwitchStatement 35 }); ContinueStatement 25 }); DoWhileStatement 12 LabeledStatement 6 http://ariya.ofilabs.com/2012/04/most-popular-javascript-statements.html
  32. Identifier Length Distribution 750 mean of the identifier length is 8.27 characters 500 prototype-1.7.0.0.js SCRIPT_ELEMENT_REJECTS_TEXTNODE_APPENDING prototype-1.7.0.0.js MOUSEENTER_MOUSELEAVE_EVENTS_SUPPORTED jquery-1.7.1.js subtractsBorderForOverflowNotVisible jquery.mobile-1.0.js getClosestElementWithVirtualBinding 250 prototype-1.7.0.0.js HAS_EXTENDED_CREATE_ELEMENT_SYNTAX 0 0 5 10 15 20 25 30 35 40 45 http://ariya.ofilabs.com/2012/05/javascript-identifier-length-distribution.html
  33. “Code Linting” var fs = require('fs'), esprima = require('./esprima'), files = process.argv.splice(2);   Not a strict equal files.forEach(function (filename) { var content = fs.readFileSync(filename, 'utf-8'), syntax = esprima.parse(content, { loc: true }); if (x == 9) {   // do Something JSON.stringify(syntax, function (key, value) { } if (key === 'test' && value.operator === '==') console.log('Line', value.loc.start.line); return value; }); }); Another example: next-generation JSHint https://github.com/jshint/jshint-next
  34. Strict Mode Checking Duplicate data property in object literal not allowed in strict mode 'use strict'; block = { color: 'blue', height: 20, width: 10, color: 'red' }; http://ariya.ofilabs.com/2012/03/strict-mode-checks-with-esprima.html
  35. http://ariya.ofilabs.com/2011/08/hall-of-api-shame-boolean-trap.html “Boolean Trap” Finder var volumeSlider = new Slider(false); Obfuscated choice component.setHidden(false); Double-negative filter.setCaseInsensitive(false); Can you make up your mind? treeItem.setState(true, false); event.initKeyEvent("keypress", true, true, null, null, The more the merrier? false, false, false, false, 9, 0);
  36. Style Formatter https://github.com/fawek/codepainter Sample code Source CodePainter Formatted source Infer coding styles Indentation Quote for string literal Whitespace
  37. https://github.com/Constellation/escodegen Rewrite and Regenerate answer = 42; var syntax = esprima.parse('answer = 42;'); syntax.body[0].expression.right.value = 1337; escodegen.generate(syntax) answer = 1337;
  38. Code Coverage https://github.com/itay/node-cover https://github.com/coveraje/coveraje x = 42; https://github.com/pmlopes/coberturajs if (false) x = -1; http://ariya.ofilabs.com/2012/03/javascript-code-coverage-and-esprima.html
  39. http://itay.github.com/snug_codecoverage_slides/ Instrumentation for Coverage { __statement_ZyyqFc(1); var a = 5; Statement var a = 5; } { __statement_ZyyqFc(2); Expression foo(); __expression_kC$jur(3), foo(); } function foo() { function foo() { Block ... __block_n53cJc(1); }; ... }
  40. Non-Destructive Partial Source Modification Do not remove comments Intact Preserve indentation & other formatting Modified Add “contextual” information Inject or change function invocation
  41. String Literal Quotes May need proper escaping console.log("Hello") console.log('Hello') List of tokens [ { type: "Identifier", value: "console", range: [0, 7] }, { type: "Punctuator", value: ".", range: [7, 8] }, { type: "Identifier", value: "log", range: [8, 11] }, { type: "Punctuator", value: "(", range: [11, 12] }, { type: "String", value: ""Hello"", range: [12, 19] }, { type: "Punctuator", value: ")", range: [19, 19] } ] http://ariya.ofilabs.com/2012/02/from-double-quotes-to-single-quotes.html
  42. Tracking the Scalability Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k; } Array.prototype.swap = function (i, j) { Log({ name: 'Array.prototype.swap', lineNumber: 1, range: [23, 94] }); var k = this[i]; this[i] = this[j]; this[j] = k; } http://ariya.ofilabs.com/2012/01/scalable-web-apps-the-complexity-issue.html
  43. Execution Tracing 4640 function calls jQuery Mobile startup log https://gist.github.com/1823129 jquery.js 26 jQuery jquery.js 103 init undefined, undefined, [object Object] jquery.js 274 each (Function) jquery.js 631 each [object Object], (Function), undefined jquery.js 495 isFunction [object Object] jquery.js 512 type [object Object] jquery.mobile.js 1857 [Anonymous] jquery.mobile.js 642 [Anonymous] jquery.mobile.js 624 enableMouseBindings jquery.mobile.js 620 disableTouchBindings http://ariya.ofilabs.com/2012/02/tracking-javascript-execution-during-startup.html
  44. http://esprima.googlecode.com/git-history/harmony/demo/transpile.html Transpilation // Object literal property shorthand. function Point(x, y) { Harmony return { x, y }; } Intact // Object literal property shorthand. ES 5.1 function Point(x, y) { return { x: x, y: y }; } Inserted fragment
  45. Application Structure Class manifest Ext.define('My.sample.Person', { name: 'Mr. Unknown',   age: 42, { className: 'My.sample.Person', constructor: function(name) {}, functions: ['walk', 'run'],   properties: ['name', 'age'] walk: function(steps) {} } run: function(steps) {} }); http://www.sencha.com/learn/sencha-class-system/
  46. Code Outline Eclipse Functions Variables
  47. Content Assist (aka Autocomplete, aka “Intellisense”) Eclipse http://contraptionsforprogramming.blogspot.com/2012/02/better-javascript-content-assist-in.html
  48. Copy Paste Mistake function inside(point, rect) { return (point.x >= rect.x1) && (point.y >= rect.y1) && (point.x <= rect.x2) && (point.y <= rect.y1); } Wrong check
  49. Refactoring Helper // Add two numbers // Add two numbers function add(firt, two) { function add(first, two) { return firt + two; return first + two; } }
  50. Future Tree traversal Scope analysis “Richer” syntax tree Syntax transformation Pattern Matching
  51. Conclusion
  52. Smart editing Source transformation Instrumentation Modern Minification & obfuscation Parser Code coverage Documentation generator Conditional contracts Dependency analysis
  53. Thank You! ariya.hidayat@gmail.com ariya.ofilabs.com @AriyaHidayat
Advertisement