Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

JavaScript and the AST Slide 1 JavaScript and the AST Slide 2 JavaScript and the AST Slide 3 JavaScript and the AST Slide 4 JavaScript and the AST Slide 5 JavaScript and the AST Slide 6 JavaScript and the AST Slide 7 JavaScript and the AST Slide 8 JavaScript and the AST Slide 9 JavaScript and the AST Slide 10 JavaScript and the AST Slide 11 JavaScript and the AST Slide 12 JavaScript and the AST Slide 13 JavaScript and the AST Slide 14 JavaScript and the AST Slide 15 JavaScript and the AST Slide 16 JavaScript and the AST Slide 17 JavaScript and the AST Slide 18 JavaScript and the AST Slide 19 JavaScript and the AST Slide 20 JavaScript and the AST Slide 21 JavaScript and the AST Slide 22 JavaScript and the AST Slide 23 JavaScript and the AST Slide 24 JavaScript and the AST Slide 25 JavaScript and the AST Slide 26 JavaScript and the AST Slide 27 JavaScript and the AST Slide 28 JavaScript and the AST Slide 29 JavaScript and the AST Slide 30 JavaScript and the AST Slide 31 JavaScript and the AST Slide 32 JavaScript and the AST Slide 33 JavaScript and the AST Slide 34 JavaScript and the AST Slide 35 JavaScript and the AST Slide 36 JavaScript and the AST Slide 37 JavaScript and the AST Slide 38 JavaScript and the AST Slide 39 JavaScript and the AST Slide 40 JavaScript and the AST Slide 41 JavaScript and the AST Slide 42 JavaScript and the AST Slide 43 JavaScript and the AST Slide 44 JavaScript and the AST Slide 45 JavaScript and the AST Slide 46 JavaScript and the AST Slide 47 JavaScript and the AST Slide 48 JavaScript and the AST Slide 49 JavaScript and the AST Slide 50 JavaScript and the AST Slide 51 JavaScript and the AST Slide 52 JavaScript and the AST Slide 53 JavaScript and the AST Slide 54 JavaScript and the AST Slide 55 JavaScript and the AST Slide 56 JavaScript and the AST Slide 57 JavaScript and the AST Slide 58 JavaScript and the AST Slide 59 JavaScript and the AST Slide 60 JavaScript and the AST Slide 61 JavaScript and the AST Slide 62 JavaScript and the AST Slide 63 JavaScript and the AST Slide 64 JavaScript and the AST Slide 65 JavaScript and the AST Slide 66 JavaScript and the AST Slide 67 JavaScript and the AST Slide 68 JavaScript and the AST Slide 69 JavaScript and the AST Slide 70 JavaScript and the AST Slide 71 JavaScript and the AST Slide 72 JavaScript and the AST Slide 73 JavaScript and the AST Slide 74 JavaScript and the AST Slide 75 JavaScript and the AST Slide 76 JavaScript and the AST Slide 77 JavaScript and the AST Slide 78 JavaScript and the AST Slide 79 JavaScript and the AST Slide 80 JavaScript and the AST Slide 81 JavaScript and the AST Slide 82 JavaScript and the AST Slide 83 JavaScript and the AST Slide 84
Upcoming SlideShare
AST - the only true tool for building JavaScript
Next
Download to read offline and view in fullscreen.

4 Likes

Share

Download to read offline

JavaScript and the AST

Download to read offline

This was a talk given at HTML5DevConf SF in 2015.
Ever wanted to write your own Browserify or Babel? Maybe have an idea for something new? This talk will get you started understanding how to use a JavaScript AST to transform and generate new code.

Related Books

Free with a 30 day trial from Scribd

See all

JavaScript and the AST

  1. 1. JavaScript and the AST Jarrod Overson - Shape Security @jsoverson
  2. 2. var jQuery = require('jquery')
  3. 3. (function(){ //browserify stuff })({ 1: [function(require, module, exports) { var jQuery = require('jquery'); }, { "jquery": 2 }], 2: [function(require, module, exports) { //*! jQuery v1.11.3 | (c) 2005, 2015 jQuery !function(a,b){"object"==typeof module&&"object }, {}] }, {}, [1]);
  4. 4. (function(){ //browserify stuff })({ 1: [function(require, module, exports) { var jQuery = require('jquery'); }, { "jquery": 2 }], 2: [function(require, module, exports) { //*! jQuery v1.11.3 | (c) 2005, 2015 jQuery !function(a,b){"object"==typeof module&&"object }, {}] }, {}, [1]);
  5. 5. (function(){ //browserify stuff })({ 1: [function(require, module, exports) { var jQuery = require('jquery'); }, { "jquery": 2 }], 2: [function(require, module, exports) { //*! jQuery v1.11.3 | (c) 2005, 2015 jQuery !function(a,b){"object"==typeof module&&"object }, {}] }, {}, [1]);
  6. 6. Less this
  7. 7. More this
  8. 8. var jQuery = require('jquery')
  9. 9. // var jQuery = require('jquery')
  10. 10. /* // */ var jQuery = require('jquery');
  11. 11. var jQuery = "require('jquery')"
  12. 12. var module = 'jquery'; var jQuery = require(module)
  13. 13. files.map(file => file.ext))
  14. 14. files.map(function (file) { return file.ext; }) files.map(file => file.ext))
  15. 15. files.map(file => this.parse(file)); var _this = this; files.map(function (file) { return _this.parse(file); });
  16. 16. Let's say you wanted to tweak your code. Just a little bit.
  17. 17. foo(a,b) Maybe… bar(b,a)
  18. 18. Meh. I got RegExes.
  19. 19. Remember life before RegExes? Has anyone thought : "Meh, I got text search."
  20. 20. /foo()/ foo()
  21. 21. foo(a) /foos*((:?[_$a-zA-Z][_$a-zA- Z0-9]*|,)*?)/
  22. 22. foo(ಠ_ಠ) /foo(([_$a-zA-ZxA0-uFFFF][_ $a-zA-Z0-9xA0-uFFFF]*)?)/
  23. 23. foo(4) /foo((:?d*|[_$a-zA-ZxA0- uFFFF][_$a-zA-Z0-9xA0- uFFFF]*)?)/
  24. 24. foo(4,a) /foo((:?d*|[_$a-zA-ZxA0- uFFFF][_$a-zA-Z0-9xA0- uFFFF]*|,)*?)/
  25. 25. foo (4,a) /foos*((:?d*|[_$a-zA-ZxA0- uFFFF][_$a-zA-Z0-9xA0- uFFFF]*|,)*?)/
  26. 26. otherfoo() bfoos*((:?d*|[_$a-zA-ZxA0- uFFFF][_$a-zA-Z0-9xA0- uFFFF]*|,)*?)
  27. 27. foo("2") /bfoos*((:?(['"])[^2]*2| d*|[_$a-zA-ZxA0-uFFFF][_$a- zA-Z0-9xA0-uFFFF]*|,)*?)/
  28. 28. foo(''.prototype.toUpperCase.call([a ,""",b,"""].join(''))) /(╯°□°)╯︵ ┻━┻/
  29. 29. Using esquery: [callee.name="foo"]
  30. 30. Ok. Ok. What's an AST?
  31. 31. Abstract Syntax Tree A tree representation of the syntactic structure of source code.
  32. 32. • How do you get a JavaScript AST? • How do you use it? • How do you transform it? • Building a transpiler.
  33. 33. Parsing JavaScript • Esprima - Fast, conservative. Parses to ESTree format. • Acorn - Error tolerant. Parses to ESTree format. • Shift - Most spec-compliant. Parses to Shift format.
  34. 34. ESTree Shift • Community effort • Wide tool support • Somewhat backwards 
 compatible with 
 SpiderMonkey AST • Shape Security product • Limited tool support • Not compatible with ESTree • Cross platform • First spec-based AST vs Standardized AST formats
  35. 35. Why ever use Shift? • Shift was developed by Ariya Hidayat (Esprima), Michael Ficarra (CoffeeScript, loads of ES tools), and several others. • Shift was designed specifically for simpler and more efficient transformation and analysis code. • Shift creators still contribute to ESTree. • When in doubt, use ESTree for the community. • If ESTree causes problems, consider Shift.
  36. 36. let ast = parse(source);
  37. 37. Just a Plain Old JavaScript Object { "type": "Script", "directives": [], "statements": [ { "type": "VariableDeclarationStatement", "declaration": { "type": "VariableDeclaration", "kind": "var", "declarators": [ { "type": "VariableDeclarator", "binding": { "type": "BindingIdentifier", "name": "a" }, "init": { "type": "CallExpression", "callee": { "type": "FunctionExpression", "isGenerator": false, "name": null, "params": { "type": "FormalParameters", "items": [ { "type": "BindingIdentifier", "name": "a" } ],
  38. 38. AST Explorer http://felix-kling.de/esprima_ast_explorer/
  39. 39. Ready made traversal tools/helpers • estraverse (estree) • esprima-walk (estree) • ast-traverse (estree) • shift-traverse (shift) • shift-reducer (shift)
  40. 40. How do you transform an AST? let a = 2; let b = 2;
  41. 41. How do you transform an AST? { type: 'Script', directives: [], statements: [ { type: 'VariableDeclarationStatement', declaration: { type: 'VariableDeclaration', kind: 'let', declarators: [ { type: 'VariableDeclarator', binding: { type: 'BindingIdentifier', name: 'a' }, init: { type: 'LiteralNumericExpression', value: 2 }
  42. 42. How do you transform an AST? import {parseScript} from 'shift-parser'; import codegen from 'shift-codegen'; let ast = parse('let a = 2;'); ast. statements[0]. declaration. declarators[0]. binding.name = 'b'; let newSource = codegen(ast);
  43. 43. OK, Let's build a transpiler. I really really like Arrow Expressions, how 'bout you?
  44. 44. Test first: what are we expecting? (() => { return 2 })() If we run* the following in an ES5 environment it will return 2 * transpile and eval()
  45. 45. shift-reducer Like [].reduce() but for ASTs
  46. 46. import reduce from 'shift-reducer'; var result = reduce(reducer, ast);
  47. 47. import spec from 'shift-spec'; let reducer = {}; for (var nodeName in spec) { reducer["reduce" + nodeName] = function (node, state) { return state; }; }
  48. 48. Hypothesis: (function(){ return 2 })() Transforming the ArrowExpression into a FunctionExpression, eg should pass the test.
  49. 49. reducer.reduceArrowExpression = function(node, state) { state.type = 'FunctionExpression'; return state; }
  50. 50. (function(){return 2}()) (()=>{return 2})()
  51. 51. Easy peasy.
  52. 52. Next step, omit the return. (() => 2)()
  53. 53. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [], statements : [{ type: 'ReturnStatement', expression: oldBody }] } } return state; }
  54. 54. (function(){return 2}()) (() => 2)()
  55. 55. And now for this () => this.foo
  56. 56. The good way… // assign "this" to a temporary variable var _this = this; // before we access it here function(){ return _this.foo }
  57. 57. The easy way… function(){ return this.foo }.bind(this) (The good way is left as an exercise to you, mostly due to time)
  58. 58. ArrowExpressions && bind() this.val = 2; arrow = () => this.val; arrow() arrow.bind({val:6})() obj = { val : 12, arrow : arrow } obj.arrow(); // 2 // 2 // 2
  59. 59. FunctionExpressions && bind() this.val = 2; fn = function() { return this.val }; fn() fn.bind({val:6})() obj = { val : 12, fn : fn } obj.fn(); // 2 // 6 // 12
  60. 60. function(){ return this.foo }.bind(this) FunctionExpression
  61. 61. function(){ return this.foo }.bind(this) CallExpression FunctionExpression
  62. 62. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [],()
  63. 63. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [],<state>.bind()
  64. 64. { type: "CallExpression", callee: { type: "StaticMemberExpression", object: { type: "IdentifierExpression", name: "foo" }, property: "bind" }, arguments: [] } foo.bind() AST BREAK!
  65. 65. { type: "CallExpression", callee: { type: "ComputedMemberExpression", object: { type:"IdentifierExpression", name:"foo" }, expression: { type: "LiteralStringExpression", value: "bind" } }, arguments: [] } foo["bind"]() AST BREAK!
  66. 66. { type: "CallExpression", callee: { type: "IdentifierExpression", name: "foo" }, arguments: [] } foo() AST BREAK!
  67. 67. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [],<state>.bind()
  68. 68. reduceArrowExpression(node, state) { state.type = 'FunctionExpression'; let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [],<state>.bind(this)
  69. 69. }, arguments: [{ type: 'ThisExpression' }] }; if (state.body.type !== 'FunctionBody') { var oldBody = state.body; state.body = { type : 'FunctionBody', directives : [], statements : [{ type: 'ReturnStatement', expression: oldBody ] } } return callExpression; }
  70. 70. "use strict"; import spec from 'shift-spec'; let reducer = {}; for (var nodeName in spec) { reducer['reduce' + nodeName] = (node, state) => state; } reducer.reduceArrowExpression = function (node, state) { let callExpression = { type: 'CallExpression', callee: { type: 'StaticMemberExpression', object: state, property: 'bind' }, arguments: [{type: 'ThisExpression'}] }; state.type = 'FunctionExpression'; if (state.body.type !== 'FunctionBody') { let oldBody = state.body; state.body = { type: 'FunctionBody', directives: [], statements: [ { type: 'ReturnStatement', expression: oldBody } ] } } return callExpression; }; export default reducer;
  71. 71. import {parseScript} from 'shift-parser'; import reduce from 'shift-reducer'; import codegen from 'shift-codegen'; import transpiler from './transpiler'; let ast = parseScript(source); var result = reduce(transpiler, ast); var newSource = codegen(result); console.log(newSource);
  72. 72. Neato! Now what?
  73. 73. Analyze Transform • Linting • Complexity • Auto Documentation • Type Checking • API Conformance • Transpiling • Code generation • Preprocessing • Refactoring • Reformatting What can you do with an AST?
  74. 74. "The worst mistake man ever committed was treating source code as text." - John Stamos Season 3, episode 20 of "Full House"
  75. 75. When have you ever typed this function greet(target) { } without this?
  76. 76. Have you ever been concerned about this string? function greet(target) { }
  77. 77. JavaScript and the AST Jarrod Overson - Shape Security @jsoverson
  • GarriotZhang

    Oct. 20, 2015
  • MichaelMerriam1

    Oct. 20, 2015
  • ryotasd

    Oct. 20, 2015
  • AndrAugusto10

    Oct. 19, 2015

This was a talk given at HTML5DevConf SF in 2015. Ever wanted to write your own Browserify or Babel? Maybe have an idea for something new? This talk will get you started understanding how to use a JavaScript AST to transform and generate new code.

Views

Total views

2,289

On Slideshare

0

From embeds

0

Number of embeds

39

Actions

Downloads

26

Shares

0

Comments

0

Likes

4

×