James Titcumb, profile picture
Uploaded byJames Titcumb
239 views

Climbing the Abstract Syntax Tree (Southeast PHP 2018)

The document discusses the intricacies of the PHP lexer and parser, particularly focusing on the abstract syntax tree (AST) and how PHP code is parsed and executed. It outlines various components including conditional statements and demonstrates the AST representation of different PHP expressions. The content emphasizes the efficiency and structure provided by the AST in parsing PHP code.

Embed presentation

Download to read offline
1 / 123
2 / 123
3 / 123
4 / 123
5 / 123
6 / 123
7 / 123
8 / 123
9 / 123
10 / 123
11 / 123
12 / 123
13 / 123
14 / 123
15 / 123
16 / 123
17 / 123
18 / 123
19 / 123
20 / 123
21 / 123
22 / 123
23 / 123
24 / 123
25 / 123
26 / 123
27 / 123
28 / 123
29 / 123
30 / 123
31 / 123
32 / 123
33 / 123
34 / 123
35 / 123
36 / 123
37 / 123
38 / 123
39 / 123
40 / 123
41 / 123
42 / 123
43 / 123
44 / 123
45 / 123
46 / 123
47 / 123
48 / 123
49 / 123
50 / 123
51 / 123
52 / 123
53 / 123
54 / 123
55 / 123
56 / 123
57 / 123
58 / 123
59 / 123
60 / 123
61 / 123
62 / 123
63 / 123
64 / 123
65 / 123
66 / 123
67 / 123
68 / 123
69 / 123
70 / 123
71 / 123
72 / 123
73 / 123
74 / 123
75 / 123
76 / 123
77 / 123
78 / 123
79 / 123
80 / 123
81 / 123
82 / 123
83 / 123
84 / 123
85 / 123
86 / 123
87 / 123
88 / 123
89 / 123
90 / 123
91 / 123
92 / 123
93 / 123
94 / 123
95 / 123
96 / 123
97 / 123
98 / 123
99 / 123
100 / 123
101 / 123
102 / 123
103 / 123
104 / 123
105 / 123
106 / 123
107 / 123
108 / 123
109 / 123
110 / 123
111 / 123
112 / 123
113 / 123
114 / 123
115 / 123
116 / 123
117 / 123
118 / 123
119 / 123
120 / 123
121 / 123
122 / 123
123 / 123
@asgrim Climbing the Abstract Syntax Tree James Titcumb Southeast PHP 2018
$ whoami James Titcumb www.jamestitcumb.com www.roave.com @asgrim
@asgrim How PHP works PHP code OpCache Execute (VM) Lexer + Parser Compiler
@asgrim The PHP Lexer zend_language_scanner.l
@asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
@asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
@asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
@asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function" { RETURN_TOKEN(T_FUNCTION); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng - 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }
@asgrim The PHP Lexer zend_language_scanner.l
@asgrim The PHP Lexer zend_language_scanner.l re2c
@asgrim The PHP Lexer zend_language_scanner.l re2c zend_language_scanner.c
@asgrim The PHP Parser zend_language_parser.y
@asgrim zend_language_parser.y if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ;
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y
@asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse
@asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A)
@asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A) if_stmt_without_else (B)
@asgrim if ($a == 1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A) if_stmt_without_else (B) if_stmt
@asgrim Zend_language_parser.y (PHP 7.0.10) if_stmt: if_stmt_without_else %prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ;
@asgrim zend_language_parser.y (PHP 5.6.26) T_IF parenthesis_expr { zend_do_if_cond(&$2, &$1 TSRMLS_CC); } statement { zend_do_if_after_statement(&$1, 1 TSRMLS_CC); } void zend_do_if_cond(const znode *cond, znode *closing_bracket_token TSRMLS_DC) { int if_cond_op_number = get_next_op_number(CG(active_op_array)); zend_op *opline = get_next_op(CG(active_op_array) TSRMLS_CC); opline->opcode = ZEND_JMPZ; SET_NODE(opline->op1, cond); closing_bracket_token->u.op.opline_num = if_cond_op_number; SET_UNUSED(opline->op2); INC_BPC(CG(active_op_array)); }
@asgrim AST is new in PHP 7+
@asgrim How PHP works PHP code OpCache Execute (VM) Lexer + Parser Compiler
@asgrim Let’s simplify!
@asgrim First… WTF is AST?
@asgrim AST is just a data structure
@asgrim PHP code <?php echo "Hello world";
@asgrim An AST representation Echo statement `-- String, value "Hello world"
@asgrim PHP code <?php echo "Hello " . "world";
@asgrim An AST representation Echo statement `-- Concat |-- Left | `-- String, value "Hello " `-- Right `-- String, value "world"
@asgrim PHP code <?php $a = 5; $b = 3; echo $a + ($b * 2);
@asgrim An AST representation Assign statement |-- Variable $a `-- Integer, value 5 Assign statement |-- Variable $b `-- Integer, value 3 Echo statement `-- Add operation |-- Left | `-- Variable $a `-- Right `-- Multiply operation |-- Left | `-- Variable $b `-- Right `-- Integer, value 2
@asgrim Why?
@asgrim Faster!*
@asgrim AST compilation Statements EchoAssign Scalar value: (int)5 Variable name: $a Assign Scalar value: (int)3 Variable name: $b Add op Right operandLeft operand Variable name: $a Multiply op Right operandLeft operand Variable name: $b Scalar value: (int)2
@asgrim AST compilation: pre-order traversal Statements EchoAssign Scalar value: (int)5 Variable name: $a Assign Scalar value: (int)3 Variable name: $b Add op Right operandLeft operand Variable name: $a Multiply op Right operandLeft operand Variable name: $b Scalar value: (int)2
@asgrim Pre-order traversal: Polish notation Assign(Variable $a, Scalar 5) Assign(Variable $b, Scalar 3) Echo ( Add( Variable $a, Multiply( $b, 2 ) ) )
@asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9?
@asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3
@asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3 Operator Left operand Right operand
@asgrim Order of precedence 1 + 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3 Operator Left operand Right operand Operator Left operand Right operand
@asgrim Reverse Polish Notation 1 2 3 * +
@asgrim Reverse Polish Notation 1 2 3 * + The stack
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 2
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 2 3
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 2 3
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 2 3
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 6
@asgrim Reverse Polish Notation 1 2 3 * + The stack 1 6
@asgrim Reverse Polish Notation 1 2 3 * + The stack 7
@asgrim Let’s write a compiler (!!!) In three easy steps…
@asgrim Warning: do not use in production
@asgrim View > Source https://github.com/asgrim/basic-maths-compiler
@asgrim Define the language Tokens ● T_ADD (+) ● T_SUBTRACT (-) ● T_MULTIPLY (/) ● T_DIVIDE (*) ● T_INTEGER (d) ● T_WHITESPACE (s+)
@asgrim Step 1: Writing a simple lexer
@asgrim Using regular expressions private static $matches = [ '/^(+)/' => Token::T_ADD, '/^(-)/' => Token::T_SUBTRACT, '/^(*)/' => Token::T_MULTIPLY, '/^(/)/' => Token::T_DIVIDE, '/^(d+)/' => Token::T_INTEGER, '/^(s+)/' => Token::T_WHITESPACE, ];
@asgrim Step through the input string public function __invoke(string $input) : array { $tokens = []; $offset = 0; while ($offset < strlen($input)) { $focus = substr($input, $offset); $result = $this->match($focus); $tokens[] = $result; $offset += strlen($result->getLexeme()); } return $tokens; }
@asgrim The matching method private function match(string $input) : Token { foreach (self::$matches as $pattern => $token) { if (preg_match($pattern, $input, $matches)) { return new Token($token, $matches[1]); } } throw new RuntimeException(sprintf( 'Unmatched token, next 15 chars were: %s', substr($input, 0, 15) )); }
@asgrim Step 2: Parsing the tokens
@asgrim Order tokens by operator precedence /** * Higher number is higher precedence. * @var int[] */ private static $operatorPrecedence = [ Token::T_SUBTRACT => 0, Token::T_ADD => 1, Token::T_DIVIDE => 2, Token::T_MULTIPLY => 3, ];
@asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
@asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
@asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
@asgrim Order tokens by operator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }
@asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
@asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
@asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
@asgrim Order tokens by operator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }
@asgrim Order tokens by operator precedence // Clean up by moving any remaining operators onto the token stack while (count($operators)) { $stack[] = array_pop($operators); } return $stack;
@asgrim Order tokens by operator precedence 1 + 2 * 3 Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 + Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 + Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 + * Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 3 + * Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 3 * + * Output stack Operator stack
@asgrim Order tokens by operator precedence 1 + 2 * 3 1 2 3 * + + Output stack Operator stack
@asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
@asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
@asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
@asgrim Create AST while ($ip < count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }
@asgrim Create AST NodeBinaryOpAdd ( NodeScalarIntegerValue(1), NodeBinaryOpMultiply ( NodeScalarIntegerValue(2), NodeScalarIntegerValue(3) ) )
@asgrim Step 3: Executing the AST
@asgrim Compile & execute AST private function compileNode(NodeInterface $node) { if ($node instanceof NodeBinaryOpAbstractBinaryOp) { return $this->compileBinaryOp($node); } if ($node instanceof NodeScalarIntegerValue) { return $node->getValue(); } }
@asgrim Compile & execute AST private function compileBinaryOp(NodeBinaryOpAbstractBinaryOp $node) { $left = $this->compileNode($node->getLeft()); $right = $this->compileNode($node->getRight()); switch (get_class($node)) { case NodeBinaryOpAdd::class: return $left + $right; case NodeBinaryOpSubtract::class: return $left - $right; case NodeBinaryOpMultiply::class: return $left * $right; case NodeBinaryOpDivide::class: return $left / $right; } }
@asgrim What does this mean for me?
@asgrim AST in userland
@asgrim php-ast extension https://github.com/nikic/php-ast
@asgrim php-ast example usage <?php require 'path/to/util.php'; $code = <<<'EOC' <?php $var = 42; EOC; echo ast_dump(astparse_code($code, $version=35)), "n"; // Output: AST_STMT_LIST 0: AST_ASSIGN var: AST_VAR name: "var" expr: 42
@asgrim astkit https://github.com/sgolemon/astkit
@asgrim astkit example usage $if = AstKit::parseString(<<<EOD if (true) { echo "This is a triumph.n"; } else { echo "The cake is a lie.n"; } EOD ); $if->execute(); // First run, program is as-seen above $const = $if->getChild(0)->getChild(0); // Replace the "true" constant in the condition with false $const->graft(0, false); // Can also graft other AstKit nodes, instead of constants $if->execute(); // Second run now takes the else path
@asgrim PhpParser https://github.com/nikic/PHP-Parser
@asgrim PHP Parser <?php use PhpParserParserFactory; $parser = (new ParserFactory) ->create(ParserFactory::PREFER_PHP7); print_r($parser->parse( file_get_contents('ast-demo-src.php') ));
@asgrim Better Reflection https://github.com/Roave/BetterReflection
@asgrim Better Reflection workflow Reflector Source Locator PhpParser Reflection
@asgrim PHP Reflection $reflection = new ReflectionClass( MyExampleClass::class ); $this->assertSame( 'ExampleClass', $reflection->getShortName() );
@asgrim Better Reflection $reflection = (new BetterReflection()) ->classReflector() ->reflect(MyExampleClass::class); $this->assertSame( 'ExampleClass', $reflection->getShortName() );
@asgrim Class BetterReflection public function sourceLocator() : SourceLocator { $astLocator = $this->astLocator(); return $this->sourceLocator ?? $this->sourceLocator = new MemoizingSourceLocator(new AggregateSourceLocator([ new PhpInternalSourceLocator($astLocator), new EvaledCodeSourceLocator($astLocator), new AutoloadSourceLocator($astLocator), ])); } public function classReflector() : ClassReflector { return $this->classReflector ?? $this->classReflector = new ClassReflector($this->sourceLocator()); }
@asgrim Given a class structure... <?php class Foo { private $bar; public function thing() { } }
@asgrim … we get the AST! Class, name Foo |-- Statements | |-- Property, name bar | | |-- Type [private] | | `-- Attributes [start line: 7, end line: 9] | `-- Method, name thing | |-- Type [public] | |-- Parameters [...] | |-- Statements [...] | `-- Attributes [start line: 7, end line: 9] `-- Attributes [start line: 3, end line: 10]
@asgrim What can I use Better Reflection for?
@asgrim Monkey patching example class MyClass { public function foo() { return 5; } }
@asgrim Monkey patching example use RoaveBetterReflectionReflectorClassReflector; use RoaveBetterReflectionSourceLocatorTypeSingleFileSourceLocator; use RoaveBetterReflectionUtilAutoloadClassLoader; use RoaveBetterReflectionUtilAutoloadClassLoaderMethodFileCacheLoader; $loader = new ClassLoader(FileCacheLoader::defaultFileCacheLoader(__DIR__)); // Create the reflection first (without loading) $classInfo = (new ClassReflector( new SingleFileSourceLocator( __DIR__ . '/MyClass.php', (new BetterReflection())->astLocator() ) ))->reflect('MyClass'); $loader->addClass($classInfo);
@asgrim Monkey patching example use RoaveBetterReflectionReflectorClassReflector; use RoaveBetterReflectionSourceLocatorTypeSingleFileSourceLocator; use RoaveBetterReflectionUtilAutoloadClassLoader; use RoaveBetterReflectionUtilAutoloadClassLoaderMethodFileCacheLoader; $loader = new ClassLoader(FileCacheLoader::defaultFileCacheLoader(__DIR__)); // Create the reflection first (without loading) $classInfo = (new ClassReflector( new SingleFileSourceLocator( __DIR__ . '/MyClass.php', (new BetterReflection())->astLocator() ) ))->reflect('MyClass'); $loader->addClass($classInfo);
@asgrim Monkey patching example // Override the body...! $classInfo->getMethod('foo')->setBodyFromClosure( function () { return 4; } ); $c = new MyClass(); echo $c->foo() . "n";
@asgrim Monkey patching example // Override the body...! $classInfo->getMethod('foo')->setBodyFromClosure( function () { return 4; } ); $c = new MyClass(); echo $c->foo() . "n"; // returns 4, not 5
@asgrim To summarise ● For PHP engine: ○ AST is an efficient data structure to represent code ○ AST means faster compilation (ignoring opcache) ○ Separation in PHP engine for parser and compiler ○ https://wiki.php.net/rfc/abstract_syntax_tree ● Concepts can be used in userland ○ PHP Parser library - https://github.com/nikic/php-parser ○ Better Reflection - https://github.com/Roave/BetterReflection ■ Reflect on not-yet-loaded files ■ Monkey patching in userland code (!) ○ Static analysis opportunities ■ Better Reflection ■ Exakat static analysis (uses own AST) ■ Phan (uses php-ast extension)
Any questions? Please leave feedback! https://joind.in/talk/72e38 James Titcumb @asgrim

More Related Content

PDF
Climbing the Abstract Syntax Tree (PHP UK 2018)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (Midwest PHP 2020)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (PHP Russia 2019)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (Forum PHP 2017)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (ScotlandPHP 2018)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (DPC 2017)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (IPC Fall 2017)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (CodeiD PHP Odessa 2017)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (PHP UK 2018)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (Midwest PHP 2020)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (PHP Russia 2019)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (Forum PHP 2017)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (ScotlandPHP 2018)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (DPC 2017)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (IPC Fall 2017)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (CodeiD PHP Odessa 2017)
byJames Titcumb
 

What's hot

PDF
Climbing the Abstract Syntax Tree (PHP Developer Days Dresden 2018)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (php[world] 2019)
byJames Titcumb
 
PDF
Interpret this... (PHPem 2016)
byJames Titcumb
 
PDF
Climbing the Abstract Syntax Tree (phpDay 2017)
byJames Titcumb
 
PPT
Class 4 - PHP Arrays
byAhmed Swilam
 
PDF
Sorting arrays in PHP
byVineet Kumar Saini
 
PPTX
Php 2
byvivlinux
 
PPT
Php Using Arrays
bymussawir20
 
PDF
Barcelona.pm Curs1211 sess01
byJavier Arturo Rodríguez
 
PDF
Dades i operadors
byAlex Muntada Duran
 
PDF
What's New in Perl? v5.10 - v5.16
byRicardo Signes
 
PDF
Arrays in PHP
byVineet Kumar Saini
 
PDF
Php array
byNikul Shah
 
PPT
Arrays in PHP
byCompare Infobase Limited
 
PDF
OSDC.TW - Gutscript for PHP haters
byLin Yo-An
 
PDF
PHP 101
byMuhammad Hijazi
 
PDF
Data Types Master
byPaolo Marcatili
 
PDF
Perl6 one-liners
byAndrew Shitov
 
PDF
PHP for Python Developers
byCarlos Vences
 
Climbing the Abstract Syntax Tree (PHP Developer Days Dresden 2018)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (php[world] 2019)
byJames Titcumb
 
Interpret this... (PHPem 2016)
byJames Titcumb
 
Climbing the Abstract Syntax Tree (phpDay 2017)
byJames Titcumb
 
Class 4 - PHP Arrays
byAhmed Swilam
 
Sorting arrays in PHP
byVineet Kumar Saini
 
Php 2
byvivlinux
 
Php Using Arrays
bymussawir20
 
Barcelona.pm Curs1211 sess01
byJavier Arturo Rodríguez
 
Dades i operadors
byAlex Muntada Duran
 
What's New in Perl? v5.10 - v5.16
byRicardo Signes
 
Arrays in PHP
byVineet Kumar Saini
 
Php array
byNikul Shah
 
Arrays in PHP
byCompare Infobase Limited
 
OSDC.TW - Gutscript for PHP haters
byLin Yo-An
 
PHP 101
byMuhammad Hijazi
 
Data Types Master
byPaolo Marcatili
 
Perl6 one-liners
byAndrew Shitov
 
PHP for Python Developers
byCarlos Vences
 

Similar to Climbing the Abstract Syntax Tree (Southeast PHP 2018)

PDF
Climbing the Abstract Syntax Tree (PHP South Africa 2017)
byJames Titcumb
 
PDF
Php engine
byjulien pauli
 
PDF
PHP Internals and Virtual Machine
byjulien pauli
 
PPTX
Php Extensions for Dummies
byElizabeth Smith
 
PDF
Quick tour of PHP from inside
byjulien pauli
 
PDF
第1回PHP拡張勉強会
byIppei Ogiwara
 
PDF
Phpをいじり倒す10の方法
byMoriyoshi Koizumi
 
PDF
Create your own PHP extension, step by step - phpDay 2012 Verona
byPatrick Allaert
 
PDF
Clear php reference
byDamien Seguy
 
PDF
Review unknown code with static analysis Zend con 2017
byDamien Seguy
 
PDF
Building Custom PHP Extensions
byAzRy LLC, Caucasus School of Technology
 
PPT
PHP-01-Overview.pptfreeforeveryonecomenow
byoliverrobertjames
 
KEY
Let's build a parser!
byBoy Baukema
 
PDF
Introduction to PHP
byBradley Holt
 
PPTX
Lexing and parsing
byElizabeth Smith
 
Climbing the Abstract Syntax Tree (PHP South Africa 2017)
byJames Titcumb
 
Php engine
byjulien pauli
 
PHP Internals and Virtual Machine
byjulien pauli
 
Php Extensions for Dummies
byElizabeth Smith
 
Quick tour of PHP from inside
byjulien pauli
 
第1回PHP拡張勉強会
byIppei Ogiwara
 
Phpをいじり倒す10の方法
byMoriyoshi Koizumi
 
Create your own PHP extension, step by step - phpDay 2012 Verona
byPatrick Allaert
 
Clear php reference
byDamien Seguy
 
Review unknown code with static analysis Zend con 2017
byDamien Seguy
 
Building Custom PHP Extensions
byAzRy LLC, Caucasus School of Technology
 
PHP-01-Overview.pptfreeforeveryonecomenow
byoliverrobertjames
 
Let's build a parser!
byBoy Baukema
 
Introduction to PHP
byBradley Holt
 
Lexing and parsing
byElizabeth Smith
 

More from James Titcumb

PDF
Living the Best Life on a Legacy Project (phpday 2022).pdf
byJames Titcumb
 
PDF
Tips for Tackling a Legacy Codebase (ScotlandPHP 2021)
byJames Titcumb
 
PDF
Best practices for crafting high quality PHP apps (Bulgaria 2019)
byJames Titcumb
 
PDF
Best practices for crafting high quality PHP apps (php[world] 2019)
byJames Titcumb
 
PDF
Crafting Quality PHP Applications (PHP Joburg Oct 2019)
byJames Titcumb
 
PDF
Best practices for crafting high quality PHP apps - PHP UK 2019
byJames Titcumb
 
PDF
Best practices for crafting high quality PHP apps (ScotlandPHP 2018)
byJames Titcumb
 
PDF
Kicking off with Zend Expressive and Doctrine ORM (PHP South Africa 2018)
byJames Titcumb
 
PDF
Best practices for crafting high quality PHP apps (PHP South Africa 2018)
byJames Titcumb
 
PDF
Crafting Quality PHP Applications (PHPkonf 2018)
byJames Titcumb
 
PDF
Best practices for crafting high quality PHP apps (PHP Yorkshire 2018)
byJames Titcumb
 
PDF
Crafting Quality PHP Applications: an overview (PHPSW March 2018)
byJames Titcumb
 
PDF
Kicking off with Zend Expressive and Doctrine ORM (PHP MiNDS March 2018)
byJames Titcumb
 
PDF
Crafting Quality PHP Applications (PHP Benelux 2018)
byJames Titcumb
 
PDF
Crafting Quality PHP Applications (ConFoo YVR 2017)
byJames Titcumb
 
PDF
Dip Your Toes in the Sea of Security (ConFoo YVR 2017)
byJames Titcumb
 
PDF
Kicking off with Zend Expressive and Doctrine ORM (ConFoo YVR 2017)
byJames Titcumb
 
PDF
Dip Your Toes in the Sea of Security (IPC Fall 2017)
byJames Titcumb
 
PDF
Dip Your Toes in the Sea of Security (PHP South Africa 2017)
byJames Titcumb
 
PDF
Get Started with RabbitMQ (CoderCruise 2017)
byJames Titcumb
 
Living the Best Life on a Legacy Project (phpday 2022).pdf
byJames Titcumb
 
Tips for Tackling a Legacy Codebase (ScotlandPHP 2021)
byJames Titcumb
 
Best practices for crafting high quality PHP apps (Bulgaria 2019)
byJames Titcumb
 
Best practices for crafting high quality PHP apps (php[world] 2019)
byJames Titcumb
 
Crafting Quality PHP Applications (PHP Joburg Oct 2019)
byJames Titcumb
 
Best practices for crafting high quality PHP apps - PHP UK 2019
byJames Titcumb
 
Best practices for crafting high quality PHP apps (ScotlandPHP 2018)
byJames Titcumb
 
Kicking off with Zend Expressive and Doctrine ORM (PHP South Africa 2018)
byJames Titcumb
 
Best practices for crafting high quality PHP apps (PHP South Africa 2018)
byJames Titcumb
 
Crafting Quality PHP Applications (PHPkonf 2018)
byJames Titcumb
 
Best practices for crafting high quality PHP apps (PHP Yorkshire 2018)
byJames Titcumb
 
Crafting Quality PHP Applications: an overview (PHPSW March 2018)
byJames Titcumb
 
Kicking off with Zend Expressive and Doctrine ORM (PHP MiNDS March 2018)
byJames Titcumb
 
Crafting Quality PHP Applications (PHP Benelux 2018)
byJames Titcumb
 
Crafting Quality PHP Applications (ConFoo YVR 2017)
byJames Titcumb
 
Dip Your Toes in the Sea of Security (ConFoo YVR 2017)
byJames Titcumb
 
Kicking off with Zend Expressive and Doctrine ORM (ConFoo YVR 2017)
byJames Titcumb
 
Dip Your Toes in the Sea of Security (IPC Fall 2017)
byJames Titcumb
 
Dip Your Toes in the Sea of Security (PHP South Africa 2017)
byJames Titcumb
 
Get Started with RabbitMQ (CoderCruise 2017)
byJames Titcumb
 

Recently uploaded

PDF
PCCC25（設立25年記念PCクラスタシンポジウム）：エヌビディア合同会社 テーマ２「NVIDIA BlueField-4 DPU」
byPC Cluster Consortium
 
PDF
[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...
byWintari Yasmin
 
PDF
MuleSoft Meetup: Dreamforce'25 Tour- Vibing With AI & Agents.pdf
byTintu Jacob Shaji
 
PDF
The partnership effect: Libraries and publishers on collaborating and thrivin...
byBookNet Canada
 
PPTX
Guardrails in Action - Ensuring Safe AI with Azure AI Content Safety.pptx
byUdaiappa Ramachandran
 
PDF
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
PDF
How Much Does It Cost To Build Software
bycollinmishell2
 
PDF
[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...
byWintari Yasmin
 
PDF
[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...
byWintari Yasmin
 
PDF
[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...
bywintari3
 
PDF
Lets Build a Serverless Function with Kiro
byChris Bingham
 
PPTX
The power of Slack and MuleSoft | Bangalore MuleSoft Meetup #60
byshyamraj55
 
PDF
The Evolving Role of the CEO in the Age of AI
byPipal Tree Services Executive Search Firm
 
PDF
10 Best Automation QA Testing Software Tools in 2025.pdf
byRohitBhandari66
 
PDF
Mastering Agentic Orchestration with UiPath Maestro | Hands on Workshop
byUiPathCommunity
 
PDF
[BDD 2025 - Mobile Development] Exploring Apple’s On-Device FoundationModels
byWintari Yasmin
 
PDF
[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...
byWintari Yasmin
 
PDF
[BDD 2025 - Mobile Development] Crafting Immersive UI with E2E and AGSL Shade...
byWintari Yasmin
 
PDF
5 Common Supply Chain Attacks and How They Work | CyberPro Magazine
byCyberPro Magazine
 
PDF
Integrating AI with Meaningful Human Collaboration
byCadabra Studio
 
PCCC25（設立25年記念PCクラスタシンポジウム）：エヌビディア合同会社 テーマ２「NVIDIA BlueField-4 DPU」
byPC Cluster Consortium
 
[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...
byWintari Yasmin
 
MuleSoft Meetup: Dreamforce'25 Tour- Vibing With AI & Agents.pdf
byTintu Jacob Shaji
 
The partnership effect: Libraries and publishers on collaborating and thrivin...
byBookNet Canada
 
Guardrails in Action - Ensuring Safe AI with Azure AI Content Safety.pptx
byUdaiappa Ramachandran
 
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
How Much Does It Cost To Build Software
bycollinmishell2
 
[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...
byWintari Yasmin
 
[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...
byWintari Yasmin
 
[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...
bywintari3
 
Lets Build a Serverless Function with Kiro
byChris Bingham
 
The power of Slack and MuleSoft | Bangalore MuleSoft Meetup #60
byshyamraj55
 
The Evolving Role of the CEO in the Age of AI
byPipal Tree Services Executive Search Firm
 
10 Best Automation QA Testing Software Tools in 2025.pdf
byRohitBhandari66
 
Mastering Agentic Orchestration with UiPath Maestro | Hands on Workshop
byUiPathCommunity
 
[BDD 2025 - Mobile Development] Exploring Apple’s On-Device FoundationModels
byWintari Yasmin
 
[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...
byWintari Yasmin
 
[BDD 2025 - Mobile Development] Crafting Immersive UI with E2E and AGSL Shade...
byWintari Yasmin
 
5 Common Supply Chain Attacks and How They Work | CyberPro Magazine
byCyberPro Magazine
 
Integrating AI with Meaningful Human Collaboration
byCadabra Studio
 

Climbing the Abstract Syntax Tree (Southeast PHP 2018)