Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Climbing the Abstract Syntax Tree (php[world] 2019) Slide 1 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 2 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 3 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 4 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 5 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 6 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 7 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 8 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 9 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 10 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 11 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 12 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 13 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 14 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 15 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 16 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 17 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 18 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 19 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 20 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 21 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 22 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 23 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 24 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 25 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 26 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 27 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 28 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 29 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 30 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 31 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 32 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 33 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 34 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 35 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 36 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 37 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 38 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 39 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 40 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 41 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 42 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 43 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 44 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 45 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 46 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 47 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 48 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 49 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 50 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 51 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 52 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 53 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 54 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 55 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 56 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 57 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 58 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 59 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 60 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 61 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 62 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 63 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 64 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 65 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 66 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 67 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 68 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 69 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 70 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 71 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 72 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 73 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 74 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 75 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 76 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 77 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 78 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 79 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 80 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 81 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 82 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 83 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 84 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 85 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 86 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 87 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 88 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 89 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 90 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 91 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 92 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 93 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 94 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 95 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 96 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 97 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 98 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 99 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 100 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 101 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 102 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 103 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 104 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 105 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 106 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 107 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 108 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 109 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 110 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 111 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 112 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 113 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 114 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 115 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 116 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 117 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 118 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 119 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 120 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 121 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 122 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 123 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 124 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 125 Climbing the Abstract Syntax Tree (php[world] 2019) Slide 126
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

Climbing the Abstract Syntax Tree (php[world] 2019)

Download to read offline

The new Abstract Syntax Tree (AST) in PHP 7 means the way our PHP code is being executed has changed. Understanding this new fundamental compilation step is key to understanding how our code is being run.

To demonstrate, James will show how a basic compiler works and how introducing an AST simplifies this process. We’ll look into how these magical time-warp techniques* can also be used in your code to introspect, analyse and modify code in a way that was never possible before.

After seeing this talk, you'll have a great insight as to the wonders of an AST, and how it can be applied to both compilers and userland code.

(*actual magic or time-warp not guaranteed)

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Climbing the Abstract Syntax Tree (php[world] 2019)

  1. 1. @asgrim Climbing the Abstract Syntax Tree
  2. 2. @asgrim Climbing the Abstract Syntax Tree James Titcumb php[world] 2019
  3. 3. $ whoami James Titcumb www.jamestitcumb.com www.roave.com @asgrim
  4. 4. @asgrim How PHP works PHP code OpCache Execute (VM) Lexer + Parser Compiler
  5. 5. @asgrim The PHP Lexer zend_language_scanner.l
  6. 6. @asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
  7. 7. @asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
  8. 8. @asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
  9. 9. @asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
  10. 10. @asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
  11. 11. @asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
  12. 12. @asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
  13. 13. @asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
  14. 14. @asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
  15. 15. @asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
  16. 16. @asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
  17. 17. @asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
  18. 18. @asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
  19. 19. @asgrim The PHP Lexer zend_language_scanner.l
  20. 20. @asgrim The PHP Lexer zend_language_scanner.l re2c
  21. 21. @asgrim The PHP Lexer zend_language_scanner.l re2c zend_language_scanner.c
  22. 22. @asgrim The PHP Parser zend_language_parser.y
  23. 23. @asgrim zend_language_parser.y if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ;
  24. 24. @asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
  25. 25. @asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
  26. 26. @asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
  27. 27. @asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
  28. 28. @asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
  29. 29. @asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
  30. 30. @asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
  31. 31. @asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse
  32. 32. @asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A)
  33. 33. @asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A) if_stmt_without_else (B)
  34. 34. @asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A) if_stmt_without_else (B) if_stmt
  35. 35. @asgrim Zend_language_parser.y (PHP 7.0.10) if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ;
  36. 36. @asgrim zend_language_parser.y (PHP 5.6.26) T_IF parenthesis_expr { zend_do_if_cond(&$2, &$1 TSRMLS_CC); } statement { zend_do_if_after_statement(&$1, 1 TSRMLS_CC); } void zend_do_if_cond(const znode *cond, znode *closing_bracket_token TSRMLS_DC) { int if_cond_op_number = get_next_op_number(CG(active_op_array)); zend_op *opline = get_next_op(CG(active_op_array) TSRMLS_CC); opline->opcode = ZEND_JMPZ; SET_NODE(opline->op1, cond); closing_bracket_token->u.op.opline_num = if_cond_op_number; SET_UNUSED(opline->op2); INC_BPC(CG(active_op_array)); }
  37. 37. @asgrim AST is new in PHP 7+
  38. 38. @asgrim How PHP works PHP code OpCache Execute (VM) Lexer + Parser Compiler
  39. 39. @asgrim Let’s simplify!
  40. 40. @asgrim First… WTF is AST?
  41. 41. @asgrim AST is just a data structure
  42. 42. @asgrim PHP code <?php echo "Hello world";
  43. 43. @asgrim An AST representation Echo statement `-- String, value "Hello world"
  44. 44. @asgrim PHP code <?php echo "Hello " . "world";
  45. 45. @asgrim An AST representation Echo statement `-- Concat |-- Left | `-- String, value "Hello " `-- Right `-- String, value "world"
  46. 46. @asgrim PHP code <?php $a = 5; $b = 3; echo $a + ($b * 2);
  47. 47. @asgrim An AST representation Assign statement |-- Variable $a `-- Integer, value 5 Assign statement |-- Variable $b `-- Integer, value 3 Echo statement `-- Add operation |-- Left | `-- Variable $a `-- Right `-- Multiply operation |-- Left | `-- Variable $b `-- Right `-- Integer, value 2
  48. 48. @asgrim Why?
  49. 49. @asgrim Faster!*
  50. 50. @asgrim AST compilation Statements EchoAssign Scalar value: (int)5 Variable name: $a Assign Scalar value: (int)3 Variable name: $b Add op Right operandLeft operand Variable name: $a Multiply op Right operandLeft operand Variable name: $b Scalar value: (int)2
  51. 51. @asgrim AST compilation: pre-order traversal Statements EchoAssign Scalar value: (int)5 Variable name: $a Assign Scalar value: (int)3 Variable name: $b Add op Right operandLeft operand Variable name: $a Multiply op Right operandLeft operand Variable name: $b Scalar value: (int)2
  52. 52. @asgrim Pre-order traversal: Polish notation Assign(Variable $a, Scalar 5) Assign(Variable $b, Scalar 3) Echo ( Add( Variable $a, Multiply( $b, 2 ) ) )
  53. 53. @asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9?
  54. 54. @asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3
  55. 55. @asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3 Operator Left operand Right operand
  56. 56. @asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3 Operator Left operand Right operand Operator Left operand Right operand
  57. 57. @asgrim Reverse Polish Notation 1 2 3 * +
  58. 58. @asgrim Reverse Polish Notation 1 2 3 * + The stack
  59. 59. @asgrim Reverse Polish Notation 1 2 3 * + The stack 1
  60. 60. @asgrim Reverse Polish Notation 1 2 3 * + The stack 2 1
  61. 61. @asgrim Reverse Polish Notation 1 2 3 * + The stack 3 2 1
  62. 62. @asgrim Reverse Polish Notation 1 2 3 * + The stack 3 2 1
  63. 63. @asgrim Reverse Polish Notation 1 2 3 * + The stack 3 2 1
  64. 64. @asgrim Reverse Polish Notation 1 2 3 * + The stack 6 1
  65. 65. @asgrim Reverse Polish Notation 1 2 3 * + The stack 6 1
  66. 66. @asgrim Reverse Polish Notation 1 2 3 * + The stack 7
  67. 67. @asgrim Let’s write a compiler (!!!) In three easy steps…
  68. 68. @asgrim Warning: do not use in production
  69. 69. @asgrim View > Source https://github.com/asgrim/basic-maths-compiler
  70. 70. @asgrim Define the language Tokens (not in order) ● T_ADD (+) ● T_SUBTRACT (-) ● T_MULTIPLY (*) ● T_DIVIDE (/) ● T_INTEGER (d+) ● T_WHITESPACE (s+)
  71. 71. @asgrim Step 1: Writing a simple lexer
  72. 72. @asgrim Using regular expressions private static $matches = [ '/^(+)/' => Token::T_ADD, '/^(-)/' => Token::T_SUBTRACT, '/^(*)/' => Token::T_MULTIPLY, '/^(/)/' => Token::T_DIVIDE, '/^(d+)/' => Token::T_INTEGER, '/^(s+)/' => Token::T_WHITESPACE, ];
  73. 73. @asgrim Step through the input string public function __invoke(string $input) : array { $tokens = []; $offset = 0; while ($offset < strlen($input)) { $focus = substr($input, $offset); $result = $this->match($focus); $tokens[] = $result; $offset += strlen($result->getLexeme()); } return $tokens; }
  74. 74. @asgrim The matching method private function match(string $input) : Token { foreach (self::$matches as $pattern => $token) { if (preg_match($pattern, $input, $matches)) { return new Token($token, $matches[1]); } } throw new RuntimeException(sprintf( 'Unmatched token, next 15 chars were: %s', substr($input, 0, 15) )); }
  75. 75. @asgrim Step 2: Parsing the tokens
  76. 76. @asgrim Order tokens by operator precedence /** * Higher number is higher precedence. * @var int[] */ private static $operatorPrecedence = [ Token::T_SUBTRACT => 0, Token::T_ADD => 1, Token::T_DIVIDE => 2, Token::T_MULTIPLY => 3, ];
  77. 77. @asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
  78. 78. @asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
  79. 79. @asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
  80. 80. @asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
  81. 81. @asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
  82. 82. @asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
  83. 83. @asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
  84. 84. @asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
  85. 85. @asgrim Order tokens by operator precedence // Clean up by moving any remaining operators onto the token stack while (count($operators)) { $stack[] = array_pop($operators); } return $stack;
  86. 86. @asgrim Order tokens by operator precedence 1 + 2 * 3 Output stack Operator stack
  87. 87. @asgrim Order tokens by operator precedence 1 + 2 * 3 1Output stack Operator stack
  88. 88. @asgrim Order tokens by operator precedence 1 + 2 * 3 1 + Output stack Operator stack
  89. 89. @asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 + Output stack Operator stack
  90. 90. @asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 + * Output stack Operator stack
  91. 91. @asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 3 + * Output stack Operator stack
  92. 92. @asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 3 * + * Output stack Operator stack
  93. 93. @asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 3 * + + Output stack Operator stack
  94. 94. @asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
  95. 95. @asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
  96. 96. @asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
  97. 97. @asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
  98. 98. @asgrim Create AST NodeBinaryOpAdd ( NodeScalarIntegerValue(1), NodeBinaryOpMultiply ( NodeScalarIntegerValue(2), NodeScalarIntegerValue(3) ) )
  99. 99. @asgrim Step 3: Executing the AST
  100. 100. @asgrim Compile & execute AST private function compileNode(NodeInterface $node) { if ($node instanceof NodeBinaryOpAbstractBinaryOp) { return $this->compileBinaryOp($node); } if ($node instanceof NodeScalarIntegerValue) { return $node->getValue(); } }
  101. 101. @asgrim Compile & execute AST private function compileBinaryOp(NodeBinaryOpAbstractBinaryOp $node) { $left = $this->compileNode($node->getLeft()); $right = $this->compileNode($node->getRight()); switch (get_class($node)) { case NodeBinaryOpAdd::class: return $left + $right; case NodeBinaryOpSubtract::class: return $left - $right; case NodeBinaryOpMultiply::class: return $left * $right; case NodeBinaryOpDivide::class: return $left / $right; } }
  102. 102. @asgrim What does this mean for me?
  103. 103. @asgrim AST in userland
  104. 104. @asgrim php-ast extension https://github.com/nikic/php-ast
  105. 105. @asgrim php-ast example usage <?php require 'path/to/util.php'; $code = <<<'EOC' <?php $var = 42; EOC; echo ast_dump(astparse_code($code, $version=35)), "n"; // Output: AST_STMT_LIST 0: AST_ASSIGN var: AST_VAR name: "var" expr: 42
  106. 106. @asgrim astkit https://github.com/sgolemon/astkit
  107. 107. @asgrim
  108. 108. @asgrim astkit example usage $if = AstKit::parseString(<<<EOD if (true) { echo "This is a triumph.n"; } else { echo "The cake is a lie.n"; } EOD ); $if->execute(); // First run, program is as-seen above $const = $if->getChild(0)->getChild(0); // Replace the "true" constant in the condition with false $const->graft(0, false); // Can also graft other AstKit nodes, instead of constants $if->execute(); // Second run now takes the else path
  109. 109. @asgrim PhpParser https://github.com/nikic/PHP-Parser
  110. 110. @asgrim PHP Parser <?php use PhpParserParserFactory; $parser = (new ParserFactory) ->create(ParserFactory::PREFER_PHP7); print_r($parser->parse( file_get_contents('ast-demo-src.php') ));
  111. 111. @asgrim Better Reflection https://github.com/Roave/BetterReflection
  112. 112. @asgrim Better Reflection workflow Reflector Source Locator PhpParser Reflection
  113. 113. @asgrim PHP Reflection $reflection = new ReflectionClass( MyExampleClass::class ); $this->assertSame( 'ExampleClass', $reflection->getShortName() );
  114. 114. @asgrim Better Reflection $reflection = (new BetterReflection()) ->classReflector() ->reflect(MyExampleClass::class); $this->assertSame( 'ExampleClass', $reflection->getShortName() );
  115. 115. @asgrim Class BetterReflection public function sourceLocator() : SourceLocator { $astLocator = $this->astLocator(); return $this->sourceLocator ?? $this->sourceLocator = new MemoizingSourceLocator(new AggregateSourceLocator([ new PhpInternalSourceLocator($astLocator), new EvaledCodeSourceLocator($astLocator), new AutoloadSourceLocator($astLocator), ])); } public function classReflector() : ClassReflector { return $this->classReflector ?? $this->classReflector = new ClassReflector($this->sourceLocator()); }
  116. 116. @asgrim Given a class structure... <?php class Foo { private $bar; public function thing() { } }
  117. 117. @asgrim … we get the AST! Class, name Foo |-- Statements | |-- Property, name bar | | |-- Type [private] | | `-- Attributes [start line: 7, end line: 9] | `-- Method, name thing | |-- Type [public] | |-- Parameters [...] | |-- Statements [...] | `-- Attributes [start line: 7, end line: 9] `-- Attributes [start line: 3, end line: 10]
  118. 118. @asgrim What can I use Better Reflection for?
  119. 119. @asgrim What is monkey patching?
  120. 120. @asgrim Monkey patching example class MyClass { public function foo() { return 5; } }
  121. 121. @asgrim Monkey patching example use RoaveBetterReflectionReflectorClassReflector; use RoaveBetterReflectionSourceLocatorTypeSingleFileSourceLocator; use RoaveBetterReflectionUtilAutoloadClassLoader; use RoaveBetterReflectionUtilAutoloadClassLoaderMethodFileCacheLoader; $loader = new ClassLoader(FileCacheLoader::defaultFileCacheLoader(__DIR__)); // Create the reflection first (without loading) $classInfo = (new ClassReflector( new SingleFileSourceLocator( __DIR__ . '/MyClass.php', (new BetterReflection())->astLocator() ) ))->reflect('MyClass'); $loader->addClass($classInfo);
  122. 122. @asgrim Monkey patching example use RoaveBetterReflectionReflectorClassReflector; use RoaveBetterReflectionSourceLocatorTypeSingleFileSourceLocator; use RoaveBetterReflectionUtilAutoloadClassLoader; use RoaveBetterReflectionUtilAutoloadClassLoaderMethodFileCacheLoader; $loader = new ClassLoader(FileCacheLoader::defaultFileCacheLoader(__DIR__)); // Create the reflection first (without loading) $classInfo = (new ClassReflector( new SingleFileSourceLocator( __DIR__ . '/MyClass.php', (new BetterReflection())->astLocator() ) ))->reflect('MyClass'); $loader->addClass($classInfo);
  123. 123. @asgrim Monkey patching example // Override the body...! $classInfo->getMethod('foo')->setBodyFromClosure( function () { return 4; } ); $c = new MyClass(); echo $c->foo() . "n";
  124. 124. @asgrim Monkey patching example // Override the body...! $classInfo->getMethod('foo')->setBodyFromClosure( function () { return 4; } ); $c = new MyClass(); echo $c->foo() . "n"; // returns 4, not 5
  125. 125. @asgrim To summarise ● For PHP engine: ○ AST is an efficient data structure to represent code ○ AST means faster compilation (ignoring opcache) ○ Separation in PHP engine for parser and compiler ○ https://wiki.php.net/rfc/abstract_syntax_tree ● Concepts can be used in userland ○ PHP Parser library - https://github.com/nikic/php-parser ○ Better Reflection - https://github.com/Roave/BetterReflection ■ Reflect on not-yet-loaded files ■ Monkey patching in userland code (!) ○ Static analysis opportunities ■ Better Reflection ■ phpstan ■ Exakat static analysis (uses own AST) ■ Phan (uses php-ast extension)
  126. 126. Any questions? https://joind.in/talk/017ce James Titcumb @asgrim

The new Abstract Syntax Tree (AST) in PHP 7 means the way our PHP code is being executed has changed. Understanding this new fundamental compilation step is key to understanding how our code is being run. To demonstrate, James will show how a basic compiler works and how introducing an AST simplifies this process. We’ll look into how these magical time-warp techniques* can also be used in your code to introspect, analyse and modify code in a way that was never possible before. After seeing this talk, you'll have a great insight as to the wonders of an AST, and how it can be applied to both compilers and userland code. (*actual magic or time-warp not guaranteed)

Views

Total views

122

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

2

Shares

0

Comments

0

Likes

0

×