Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)

1.
@asgrim Climbing the Abstract SyntaxTree James Titcumb Bulgaria PHP Conference 2016

2.
Who is thisguy? James Titcumb www.jamestitcumb.com www.roave.com www.phphants.co.uk www.phpsouthcoast.co.uk @asgrim

3.
@asgrim How PHP works PHPcode OpCache Execute (VM) Lexer + Parser Compiler

4.
@asgrim The PHP Lexer zend_language_scanner.l

5.
@asgrim zend_language_scanner.l <ST_IN_SCRIPTING>"exit" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"die" { RETURN_TOKEN(T_EXIT); } <ST_IN_SCRIPTING>"function"{ RETURN_TOKEN(T_FUNCTION); }

6.

7.

8.

9.
@asgrim zend_language_scanner.l <ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" { yy_push_state(ST_LOOKING_FOR_VARNAME); RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES); } <ST_LOOKING_FOR_VARNAME>{LABEL}[[}] { yyless(yyleng- 1); zend_copy_value(zendlval, yytext, yyleng); yy_pop_state(); yy_push_state(ST_IN_SCRIPTING); RETURN_TOKEN(T_STRING_VARNAME); }

10.

11.

12.

13.

14.

15.

16.

17.

18.
@asgrim The PHP Lexer zend_language_scanner.l

19.
@asgrim The PHP Lexer zend_language_scanner.l re2c

20.
@asgrim The PHP Lexer zend_language_scanner.l re2c zend_language_scanner.c

21.
@asgrim The PHP Parser zend_language_parser.y

22.
@asgrim zend_language_parser.y if_stmt: if_stmt_without_else %prec T_NOELSE{ $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ;

23.
@asgrim if_stmt: if_stmt_without_else %prec T_NOELSE{ $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ; zend_language_parser.y

24.

25.

26.

27.

28.

29.

30.
@asgrim if ($a ==1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse

31.
@asgrim if ($a ==1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A)

32.
@asgrim if ($a ==1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A) if_stmt_without_else (B)

33.
@asgrim if ($a ==1) { a(); } else if ($b == 1) { b(); } else { c(); } Using the rules to parse if_stmt_without_else (A) if_stmt_without_else (B) if_stmt

34.
@asgrim Zend_language_parser.y (PHP 7.0.10) if_stmt: if_stmt_without_else%prec T_NOELSE { $$ = $1; } | if_stmt_without_else T_ELSE statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); } ; if_stmt_without_else: T_IF '(' expr ')' statement { $$ = zend_ast_create_list(1, ZEND_AST_IF, zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); } | if_stmt_without_else T_ELSEIF '(' expr ')' statement { $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, $4, $6)); } ;

35.
@asgrim zend_language_parser.y (PHP 5.6.26) T_IFparenthesis_expr { zend_do_if_cond(&$2, &$1 TSRMLS_CC); } statement { zend_do_if_after_statement(&$1, 1 TSRMLS_CC); } void zend_do_if_cond(const znode *cond, znode *closing_bracket_token TSRMLS_DC) { int if_cond_op_number = get_next_op_number(CG(active_op_array)); zend_op *opline = get_next_op(CG(active_op_array) TSRMLS_CC); opline->opcode = ZEND_JMPZ; SET_NODE(opline->op1, cond); closing_bracket_token->u.op.opline_num = if_cond_op_number; SET_UNUSED(opline->op2); INC_BPC(CG(active_op_array)); }

36.
@asgrim AST is newin PHP 7+

37.
@asgrim How PHP works PHPcode OpCache Execute (VM) Lexer + Parser Compiler

38.
@asgrim Let’s simplify!

39.
@asgrim First… WTF isAST?

40.
@asgrim AST is justa data structure

41.
@asgrim PHP code <?php echo "Helloworld";

42.
@asgrim An AST representation Echostatement `-- String, value "Hello world"

43.
@asgrim PHP code <?php echo "Hello" . "world";

44.
@asgrim An AST representation Echostatement `-- Concat |-- Left | `-- String, value "Hello " `-- Right `-- String, value "world"

45.
@asgrim PHP code <?php $a =5; $b = 3; echo $a + ($b * 2);

47.
@asgrim AST compilation Statements EchoAssign Scalar value: (int)5 Variable name:$a Assign Scalar value: (int)3 Variable name: $b Add op Right operandLeft operand Variable name: $a Multiply op Right operandLeft operand Variable name: $b Scalar value: (int)2

48.
@asgrim AST compilation: pre-ordertraversal Statements EchoAssign Scalar value: (int)5 Variable name: $a Assign Scalar value: (int)3 Variable name: $b Add op Right operandLeft operand Variable name: $a Multiply op Right operandLeft operand Variable name: $b Scalar value: (int)2

49.
@asgrim Pre-order traversal: Polishnotation Assign(Variable $a, Scalar 5) Assign(Variable $b, Scalar 3) Echo ( Add( Variable $a, Multiply( $b, 2 ) ) )

50.
@asgrim Order of precedence 1+ 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9?

51.
@asgrim Order of precedence 1+ 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3

52.
@asgrim Order of precedence 1+ 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3 Operator Left operand Right operand

53.
@asgrim Order of precedence 1+ 2 * 3 = 1 + (2 * 3) = 7? = (1 + 2) * 3 = 9? + 1 * 2 3 Operator Left operand Right operand Operator Left operand Right operand

54.
@asgrim Reverse Polish Notation 12 3 * +

55.
@asgrim Reverse Polish Notation 12 3 * + The stack

56.
@asgrim Reverse Polish Notation 12 3 * + The stack 1

57.
@asgrim Reverse Polish Notation 12 3 * + The stack 1 2

58.
@asgrim Reverse Polish Notation 12 3 * + The stack 1 2 3

59.

60.

61.

62.

63.
@asgrim Reverse Polish Notation 12 3 * + The stack 7

64.
@asgrim Let’s write acompiler (!!!) In three easy steps…

65.
@asgrim Warning: do notuse in production

66.
@asgrim View > Source https://github.com/asgrim/basic-maths-compiler

67.
@asgrim Define the language Tokens ●T_ADD (+) ● T_SUBTRACT (-) ● T_MULTIPLY (/) ● T_DIVIDE (*) ● T_INTEGER (d) ● T_WHITESPACE (s+)

68.
@asgrim Step 1: Writinga simple lexer

69.
@asgrim Using regular expressions privatestatic $matches = [ '/^(+)/' => Token::T_ADD, '/^(-)/' => Token::T_SUBTRACT, '/^(*)/' => Token::T_MULTIPLY, '/^(/)/' => Token::T_DIVIDE, '/^(d+)/' => Token::T_INTEGER, '/^(s+)/' => Token::T_WHITESPACE, ];

70.
@asgrim Step through theinput string public function __invoke(string $input) : array { $tokens = []; $offset = 0; while ($offset < strlen($input)) { $focus = substr($input, $offset); $result = $this->match($focus); $tokens[] = $result; $offset += strlen($result->getLexeme()); } return $tokens; }

71.
@asgrim The matching method privatefunction match(string $input) : Token { foreach (self::$matches as $pattern => $token) { if (preg_match($pattern, $input, $matches)) { return new Token($token, $matches[1]); } } throw new RuntimeException(sprintf( 'Unmatched token, next 15 chars were: %s', substr($input, 0, 15) )); }

72.
@asgrim Step 2: Parsingthe tokens

73.
@asgrim Order tokens byoperator precedence /** * Higher number is higher precedence. * @var int[] */ private static $operatorPrecedence = [ Token::T_SUBTRACT => 0, Token::T_ADD => 1, Token::T_DIVIDE => 2, Token::T_MULTIPLY => 3, ];

74.
@asgrim Order tokens byoperator precedence /** @var Token[] $stack */ $stack = []; /** @var Token[] $operators */ $operators = []; while (false !== ($token = current($tokens))) { if ($token->isOperator()) { // ... } $stack[] = $token; next($tokens); }

75.

76.

77.

78.
@asgrim Order tokens byoperator precedence if ($token->isOperator()) { $tokenPrecedence = self::$operatorPrecedence[$token->getToken()]; while ( count($operators) && self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()] > $tokenPrecedence ) { $higherOp = array_pop($operators); $stack[] = $higherOp; } $operators[] = $token; next($tokens); continue; }

79.

80.

81.

82.
@asgrim Order tokens byoperator precedence // Clean up by moving any remaining operators onto the token stack while (count($operators)) { $stack[] = array_pop($operators); } return $stack;

83.
@asgrim Order tokens byoperator precedence 1 + 2 * 3 Output stack Operator stack

84.
@asgrim Order tokens byoperator precedence 1 + 2 * 3 1Output stack Operator stack

85.
@asgrim Order tokens byoperator precedence 1 + 2 * 3 1 + Output stack Operator stack

86.
@asgrim Order tokens byoperator precedence 1 + 2 * 3 1 2 + Output stack Operator stack

87.
@asgrim Order tokens byoperator precedence 1 + 2 * 3 1 2 + * Output stack Operator stack

88.
@asgrim Order tokens byoperator precedence 1 + 2 * 3 1 2 3 + * Output stack Operator stack

89.
@asgrim Order tokens byoperator precedence 1 + 2 * 3 1 2 3 * + * Output stack Operator stack

90.
@asgrim Order tokens byoperator precedence 1 + 2 * 3 1 2 3 * + + Output stack Operator stack

91.
@asgrim Create AST while ($ip< count($tokenStack)) { $token = $tokenStack[$ip++]; if ($token->isOperator()) { // (figure out $nodeType) $right = array_pop($astStack); $left = array_pop($astStack); $astStack[] = new $nodeType($left, $right); continue; } $astStack[] = new NodeScalarIntegerValue((int)$token->getLexeme()); }

92.

93.

94.

95.
@asgrim Create AST NodeBinaryOpAdd ( NodeScalarIntegerValue(1), NodeBinaryOpMultiply( NodeScalarIntegerValue(2), NodeScalarIntegerValue(3) ) )

96.
@asgrim Step 3: Executingthe AST

97.
@asgrim Compile & executeAST private function compileNode(NodeInterface $node) { if ($node instanceof NodeBinaryOpAbstractBinaryOp) { return $this->compileBinaryOp($node); } if ($node instanceof NodeScalarIntegerValue) { return $node->getValue(); } }

98.
@asgrim Compile & executeAST private function compileBinaryOp(NodeBinaryOpAbstractBinaryOp $node) { $left = $this->compileNode($node->getLeft()); $right = $this->compileNode($node->getRight()); switch (get_class($node)) { case NodeBinaryOpAdd::class: return $left + $right; case NodeBinaryOpSubtract::class: return $left - $right; case NodeBinaryOpMultiply::class: return $left * $right; case NodeBinaryOpDivide::class: return $left / $right; } }

99.
@asgrim AST in userland

100.
@asgrim php-ast extension https://github.com/nikic/php-ast

101.
@asgrim php-ast example usage <?php require'path/to/util.php'; $code = <<<'EOC' <?php $var = 42; EOC; echo ast_dump(astparse_code($code, $version=35)), "n"; // Output: AST_STMT_LIST 0: AST_ASSIGN var: AST_VAR name: "var" expr: 42

102.
@asgrim astkit https://github.com/sgolemon/astkit

103.
@asgrim astkit example usage $if= AstKit::parseString(<<<EOD if (true) { echo "This is a triumph.n"; } else { echo "The cake is a lie.n"; } EOD ); $if->execute(); // First run, program is as-seen above $const = $if->getChild(0)->getChild(0); // Replace the "true" constant in the condition with false $const->graft(0, false); // Can also graft other AstKit nodes, instead of constants $if->execute(); // Second run now takes the else path

104.
@asgrim PhpParser https://github.com/nikic/PHP-Parser

105.
@asgrim PHP Parser <?php use PhpParserParserFactory; $parser= (new ParserFactory) ->create(ParserFactory::PREFER_PHP7); print_r($parser->parse( file_get_contents('ast-demo-src.php') ));

106.
@asgrim Better Reflection https://github.com/Roave/BetterReflection

107.
@asgrim Better Reflection workflow Reflector Source Locator PhpParser Reflection

108.
@asgrim PHP Reflection $reflection =new ReflectionClass( MyExampleClass::class ); $this->assertSame( 'ExampleClass', $reflection->getShortName() );

109.
@asgrim Better Reflection $reflection =ReflectionClass::createFromName( MyExampleClass::class ); $this->assertSame( 'ExampleClass', $reflection->getShortName() );

110.
@asgrim ReflectionClass::createFromName() // In ReflectionClass: public static function createFromName($className) { return ClassReflector::buildDefaultReflector()->reflect($className); }

111.
@asgrim ClassReflector::buildDefaultReflector() // In ClassReflector: public static function buildDefaultReflector() { return new self(new AggregateSourceLocator([ new PhpInternalSourceLocator(), new EvaledCodeSourceLocator(), new AutoloadSourceLocator(), ])); }

112.
@asgrim Given a classstructure... <?php class Foo { private $bar; public function thing() { } }

114.
Any questions? https://joind.in/talk/513ad James Titcumb@asgrim

Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)

More Related Content

What's hot

Viewers also liked

Similar to Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)

More from James Titcumb

Recently uploaded

Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)