Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

JavaScript and the AST

1,767 views

Published on

This was a talk given at HTML5DevConf SF in 2015.
Ever wanted to write your own Browserify or Babel? Maybe have an idea for something new? This talk will get you started understanding how to use a JavaScript AST to transform and generate new code.

Published in: Technology
  • Be the first to comment

JavaScript and the AST

  1. 1. JavaScript and the AST Jarrod Overson - Shape Security @jsoverson
  2. 2. var jQuery = require('jquery')
  3. 3. (function(){ //browserify stuff })({ 1: [function(require, module, exports) { var jQuery = require('jquery'); }, { "jquery": 2 }], 2: [function(require, module, exports) { //*! jQuery v1.11.3 | (c) 2005, 2015 jQuery !function(a,b){"object"==typeof module&&"object }, {}] }, {}, [1]);
  4. 4. (function(){ //browserify stuff })({ 1: [function(require, module, exports) { var jQuery = require('jquery'); }, { "jquery": 2 }], 2: [function(require, module, exports) { //*! jQuery v1.11.3 | (c) 2005, 2015 jQuery !function(a,b){"object"==typeof module&&"object }, {}] }, {}, [1]);
  5. 5. (function(){ //browserify stuff })({ 1: [function(require, module, exports) { var jQuery = require('jquery'); }, { "jquery": 2 }], 2: [function(require, module, exports) { //*! jQuery v1.11.3 | (c) 2005, 2015 jQuery !function(a,b){"object"==typeof module&&"object }, {}] }, {}, [1]);
  6. 6. Less this
  7. 7. More this
  8. 8. var jQuery = require('jquery')
  9. 9. // var jQuery = require('jquery')
  10. 10. /* // */ var jQuery = require('jquery');
  11. 11. var jQuery = "require('jquery')"
  12. 12. var module = 'jquery'; var jQuery = require(module)
  13. 13. files.map(file => file.ext))
  14. 14. files.map(function (file) { return file.ext; }) files.map(file => file.ext))
  15. 15. files.map(file => this.parse(file)); var _this = this; files.map(function (file) { return _this.parse(file); });
  16. 16. Let's say you wanted to tweak your code. Just a little bit.
  17. 17. foo(a,b) Maybe… bar(b,a)
  18. 18. Meh. I got RegExes.
  19. 19. Remember life before RegExes? Has anyone thought : "Meh, I got text search."
  20. 20. /foo()/ foo()
  21. 21. foo(a) /foos*((:?[_$a-zA-Z][_$a-zA- Z0-9]*|,)*?)/
  22. 22. foo(ಠ_ಠ) /foo(([_$a-zA-ZxA0-uFFFF][_ $a-zA-Z0-9xA0-uFFFF]*)?)/
  23. 23. foo(4) /foo((:?d*|[_$a-zA-ZxA0- uFFFF][_$a-zA-Z0-9xA0- uFFFF]*)?)/
  24. 24. foo(4,a) /foo((:?d*|[_$a-zA-ZxA0- uFFFF][_$a-zA-Z0-9xA0- uFFFF]*|,)*?)/
  25. 25. foo (4,a) /foos*((:?d*|[_$a-zA-ZxA0- uFFFF][_$a-zA-Z0-9xA0- uFFFF]*|,)*?)/
  26. 26. otherfoo() bfoos*((:?d*|[_$a-zA-ZxA0- uFFFF][_$a-zA-Z0-9xA0- uFFFF]*|,)*?)
  27. 27. foo("2") /bfoos*((:?(['"])[^2]*2| d*|[_$a-zA-ZxA0-uFFFF][_$a- zA-Z0-9xA0-uFFFF]*|,)*?)/
  28. 28. foo(''.prototype.toUpperCase.call([a ,""",b,"""].join(''))) /(╯°□°)╯︵ ┻━┻/
  29. 29. Using esquery: [callee.name="foo"]
  30. 30. Ok. Ok. What's an AST?
  31. 31. Abstract Syntax Tree A tree representation of the syntactic structure of source code.
  32. 32. • How do you get a JavaScript AST? • How do you use it? • How do you transform it? • Building a transpiler.
  33. 33. Parsing JavaScript • Esprima - Fast, conservative. Parses to ESTree format. • Acorn - Error tolerant. Parses to ESTree format. • Shift - Most spec-compliant. Parses to Shift format.
  34. 34. ESTree Shift • Community effort • Wide tool support • Somewhat backwards 
 compatible with 
 SpiderMonkey AST • Shape Security product • Limited tool support • Not compatible with ESTree • Cross platform • First spec-based AST vs Standardized AST formats
  35. 35. Why ever use Shift? • Shift was developed by Ariya Hidayat (Esprima), Michael Ficarra (CoffeeScript, loads of ES tools), and several others. • Shift was designed specifically for simpler and more efficient transformation and analysis code. • Shift creators still contribute to ESTree. • When in doubt, use ESTree for the community. • If ESTree causes problems, consider Shift.
  36. 36. let ast = parse(source);
  37. 37. Just a Plain Old JavaScript Object { "type": "Script", "directives": [], "statements": [ { "type": "VariableDeclarationStatement", "declaration": { "type": "VariableDeclaration", "kind": "var", "declarators": [ { "type": "VariableDeclarator", "binding": { "type": "BindingIdentifier", "name": "a" }, "init": { "type": "CallExpression", "callee": { "type": "FunctionExpression", "isGenerator": false, "name": null, "params": { "type": "FormalParameters", "items": [ { "type": "BindingIdentifier", "name": "a" } ],
  38. 38. AST Explorer http://felix-kling.de/esprima_ast_explorer/
  39. 39. Ready made traversal tools/helpers • estraverse (estree) • esprima-walk (estree) • ast-traverse (estree) • shift-traverse (shift) • shift-reducer (shift)
  40. 40. How do you transform an AST? let a = 2; let b = 2;
  41. 41. How do you transform an AST? { type: 'Script', directives: [], statements: [ { type: 'VariableDeclarationStatement', declaration: { type: 'VariableDeclaration', kind: 'let', declarators: [ { type: 'VariableDeclarator', binding: { type: 'BindingIdentifier', name: 'a' }, init: { type: 'LiteralNumericExpression', value: 2 }
  42. 42. How do you transform an AST? import {parseScript} from 'shift-parser'; import codegen from 'shift-codegen'; let ast = parse('let a = 2;'); ast. statements[0]. declaration. declarators[0]. binding.name = 'b'; let newSource = codegen(ast);
  43. 43. OK, Let's build a transpiler. I really really like Arrow Expressions, how 'bout you?
  44. 44. Test first: what are we expecting? (() => { return 2 })() If we run* the following in an ES5 environment it will return 2 * transpile and eval()
  45. 45. shift-reducer Like [].reduce() but for ASTs
  46. 46. import reduce from 'shift-reducer'; var result = reduce(reducer, ast);
  47. 47. import spec from 'shift-spec'; let reducer = {}; for (var nodeName in spec) { reducer["reduce" + nodeName] = function (node, state) { return state; }; }
  48. 48. Hypothesis: (function(){ return 2 })() Transforming the ArrowExpression into a FunctionExpression, eg should pass the test.
  49. 49. reducer.reduceArrowExpression = function(node, state) { state.type = 'FunctionExpression'; return state; }
  50. 50. (function(){return 2}()) (()=>{return 2})()
  51. 51. Easy peasy.
  52. 52. Next step, omit the return. (() => 2)()
  53. 53. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [], statements : [{ type: 'ReturnStatement', expression: oldBody }] } } return state; }
  54. 54. (function(){return 2}()) (() => 2)()
  55. 55. And now for this () => this.foo
  56. 56. The good way… // assign "this" to a temporary variable var _this = this; // before we access it here function(){ return _this.foo }
  57. 57. The easy way… function(){ return this.foo }.bind(this) (The good way is left as an exercise to you, mostly due to time)
  58. 58. ArrowExpressions && bind() this.val = 2; arrow = () => this.val; arrow() arrow.bind({val:6})() obj = { val : 12, arrow : arrow } obj.arrow(); // 2 // 2 // 2
  59. 59. FunctionExpressions && bind() this.val = 2; fn = function() { return this.val }; fn() fn.bind({val:6})() obj = { val : 12, fn : fn } obj.fn(); // 2 // 6 // 12
  60. 60. function(){ return this.foo }.bind(this) FunctionExpression
  61. 61. function(){ return this.foo }.bind(this) CallExpression FunctionExpression
  62. 62. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [],()
  63. 63. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [],<state>.bind()
  64. 64. { type: "CallExpression", callee: { type: "StaticMemberExpression", object: { type: "IdentifierExpression", name: "foo" }, property: "bind" }, arguments: [] } foo.bind() AST BREAK!
  65. 65. { type: "CallExpression", callee: { type: "ComputedMemberExpression", object: { type:"IdentifierExpression", name:"foo" }, expression: { type: "LiteralStringExpression", value: "bind" } }, arguments: [] } foo["bind"]() AST BREAK!
  66. 66. { type: "CallExpression", callee: { type: "IdentifierExpression", name: "foo" }, arguments: [] } foo() AST BREAK!
  67. 67. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [],<state>.bind()
  68. 68. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [],<state>.bind(this)
  69. 69. }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [], statements : [{ type: 'ReturnStatement', expression: oldBody ] } } return callExpression; }
  70. 70. "use strict"; import spec from 'shift-spec'; let reducer = {}; for (var nodeName in spec) { reducer['reduce' + nodeName] = (node, state) => state; } reducer.reduceArrowExpression = function (node, state) { let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{type: 'ThisExpression'}] }; state.type = 'FunctionExpression'; if (state.body.type !== 'FunctionBody') { let oldBody = state.body; state.body = { type: 'FunctionBody', directives: [], statements: [ { type: 'ReturnStatement', expression: oldBody } ] } } return callExpression; }; export default reducer;
  71. 71. import {parseScript} from 'shift-parser'; import reduce from 'shift-reducer'; import codegen from 'shift-codegen'; import transpiler from './transpiler'; let ast = parseScript(source); var result = reduce(transpiler, ast); var newSource = codegen(result); console.log(newSource);
  72. 72. Neato! Now what?
  73. 73. Analyze Transform • Linting • Complexity • Auto Documentation • Type Checking • API Conformance • Transpiling • Code generation • Preprocessing • Refactoring • Reformatting What can you do with an AST?
  74. 74. "The worst mistake man ever committed was treating source code as text." - John Stamos Season 3, episode 20 of "Full House"
  75. 75. When have you ever typed this function greet(target) { } without this?
  76. 76. Have you ever been concerned about this string? function greet(target) { }
  77. 77. JavaScript and the AST Jarrod Overson - Shape Security @jsoverson

×