Today's Web Development tooling is now a very rich ecosystem that allows very professional workflows, and the fun thing is that they all share a very interesting technology called AST, for "Abstract syntax tree". That's so powerful that some people see it as a very complex magic thing. Now the fun part... It is in fact quite simple to understand., and Today's libraries (Arcon, Esprima, Espree, Babylon) allows anyone to create their own tools, fork existing ones, or write plugins matching our own desires.

1. Lint, Coverage, Doc,
Autocompletion, Transpilation,
Minification... powered by AST
Alexandre Morgaut
NantesJS Meetup #39 - April 2019

2. ● Ealy JavaScript Adopter (1996)
● Former Wakanda Server-Side JS platform Team (2008)
● JS & Web Standards Evangelist since 2009
● Former member of CommonJS (2010)
● Former W3C Advisory Committee member (2012)
● Initiated NantesJS (2012)
● Joined the Wiztivi company (2015)
Presentation
@amorgaut wiztivi.comAMorgaut

3. Summary
● Compilers
● BinAST
● Editors
● JavaScript AST
● Using AST in Plugins
● Task Managers
● AST Explorer

5. Compilers

6. http://www.cs.man.ac.uk/~pjj/farrell/comp3.html
Anatomy of a Compiler

7. http://www-igm.univ-mlv.fr/~forax/ens/java-avance/cours/pdf/old/Generation%20de%20code%201.pdf

8. Some Token Types
Name Examples
identifiers x, color, UP​
keywords if, while, return​
punctuation }, (, ;
separators +, <, =​
literals true, 6.02e23, "music"​
comments // en ligne, /* en bloc */​

9. BinAST

10. Binary AST
● Give the JS engine a pre-tokenized, pre-parsed, script content
● Give it in binary format to make it lighter
● Current advanced prototypes already show JS parsing improvements of 30%-50%
on all the most common frameworks
https://github.com/tc39/proposal-binary-ast
https://github.com/binast/binjs-ref

11. https://yoric.github.io/Fosdem-2019/

12. Editors

13. Editor Features
● With Tokenizer Help
○ Syntax highlighting
○ Collapse / Expand
● With AST Help
○ Lint
○ Find Usages
○ Code Completion
○ Reformat
○ Statement level breakpoints

14. ACE - The Cloud9 Editor
// ### Abstract Syntax Tree
//
// The default AST structure is inspired by the Mozilla Parser API but can
// easily be customized by overriding these functions.
// Lexer
// -----
//
// The lexer, or the tokenizer reads the input string character by character
// and derives a token left-right. To be as efficient as possible the lexer
// prioritizes the common cases such as identifiers. It also works with
// character codes instead of characters as string comparisons was the
// biggest bottleneck of the parser.
https://github.com/ajaxorg/ace/blob/9b5b63d1dc7c1b81b58d30c87d14b5905d030ca5/lib/ace/mode/lua/luaparse.js

15. ternjs
code-analysis engine for JavaScript
● Autocompletion on variables and properties
● Function argument hints
● Querying the type of an expression
● Finding the definition of something
● Automatic refactoring
Built-in in the Bracket Editor (Adobe)
https://ternjs.net/

16. JavaScript AST

17. SpiderMonkey engine's JavaScript parser API
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Parser_API

18. Parsers
● espree
○ ESLint, estraverse, hint, javascript-obfuscator, JSDoc, protobuf.js,
v8/web-tooling-benchmark
● babylon / babel-parser
○ babel, ESDoc
● esprima
○ previous ESLint, flow, istanbul
● acorn
○ espree, buble, browserify, ternjs, webpack, optimize-js

19. Espree
● Fork of Esprima (v1.2.2)
○ Primary goal is to produce the exact same AST structure and tokens as Esprima
○ The AST structure being the ESTree API with JSX extensions
● Built on top of Acorn
○ use its modular architecture -> extensions of core functionalities
● Maintained by the ESLint project
https://github.com/eslint/espree
const ast = espree.parse(code);

20. ESTree
"This site will serve as a community standard"
"keep up with the evolution of the JavaScript language"
https://github.com/estree/estree

21. AST Example

22. Using AST in Plugins

23. ESLint Rule
Enforce a maximum number of parameters in function definitions
https://eslint.org/docs/rules/max-params
https://github.com/eslint/eslint/blob/master/lib/rules/max-params.js

24. Rule Option Management
Rule Validation Code
AST Node Type Binding

25. Babel Transform Plugin
babel-plugin-transform-class
A minimalist ES6 class babel transformer
● Remove ES6 forbidden actions controls
○ ex: throw on this modification before calling super()
○ Encourage to check that at edit time using ESLint rules
● Remove Polyfill generations
○ Generate Full ES5 compatible code
○ Even compatible with ES3 using very minimalist Polyfills
https://github.com/AMorgaut/babel-plugin-transform-class

26. Return a visitor object
● keys:
AST Node types
● values:
callback
or map of callbacks
○ enter:
beginning of the node block
○ exit:
end of the node block
babel.types
● provides useful helpers
https://github.com/AMorgaut/babel-plugin-transform-class/blob/master/src/index.js

27. Task Managers

28. Task Manager
Grunt, Gulp, npm, ...
webpack
● $ npm run build
○ lint > transpile* > concat* > minify*
○ * with SourceMap
○ each step using it's own AST
● Some of those tools (ex: babel) can use an existing AST

29. AST Explorer

30. https://astexplorer.net
https://astexplorer.net/#/gist/0178b41edea28820b2452e3422059cbd/latest
Input
Source
Transform
Code
Input
AST
Output
Source

31. AST Explorer
1. Select (or use default) a:
a. Source Type (JavaScript, CSS, HTML, …)
b. Parser Lib (babel, eslint, …)
c. (Optional) Transformer Lib
2. Provide a Source Code
a. A Browsable matching AST is automatically generated
3. Provide a Transform Code
a. An Output Source is automatically updated
b. Error messages code be shown instead to help fixing the code

32. Questions ?

33. Thank you !