Implementing a Simple Interpreter

1. Implementing a Simple Interpreter
Ray Song
May 27, 2012
Ray Song
Implementing a Simple Interpreter

2. Flow sheet
Lexical analysis
Grammatical analysis
Figure : Flow
Ray Song
Implementing a Simple Interpreter

3. Flow sheet
Lexical analysis
Grammatical analysis
Figure : Flow
Ray Song
Implementing a Simple Interpreter

4. Lexical Analysis
manual
flex
Ray Song
Implementing a Simple Interpreter

5. Lexical Analysis
manual
flex
Ray Song
Implementing a Simple Interpreter

6. Tokens
identifier
integer
string
while
if
else
print
NEWLINE
NO WHITESPACE
Ray Song
Implementing a Simple Interpreter

7. Tokens
identifier
integer
string
while
if
else
print
NEWLINE
NO WHITESPACE
Ray Song
Implementing a Simple Interpreter

8. Tokens
identifier
integer
string
while
if
else
print
NEWLINE
NO WHITESPACE
Ray Song
Implementing a Simple Interpreter

9. Tokens
identifier
integer
string
while
if
else
print
NEWLINE
NO WHITESPACE
Ray Song
Implementing a Simple Interpreter

10. Tokens
identifier
integer
string
while
if
else
print
NEWLINE
NO WHITESPACE
Ray Song
Implementing a Simple Interpreter

11. Tokens
identifier
integer
string
while
if
else
print
NEWLINE
NO WHITESPACE
Ray Song
Implementing a Simple Interpreter

12. Tokens
identifier
integer
string
while
if
else
print
NEWLINE
NO WHITESPACE
Ray Song
Implementing a Simple Interpreter

13. Tokens
identifier
integer
string
while
if
else
print
NEWLINE
NO WHITESPACE
Ray Song
Implementing a Simple Interpreter

14. Tokens
identifier
integer
string
while
if
else
print
NEWLINE
NO WHITESPACE
Ray Song
Implementing a Simple Interpreter

15. Off-side rule
def hello():
print ‘world’
indentation sensitive
Ex: ISWIM(1966), occam(1983), Miranda(1985),
Haskell(1990), Python(1991)
Ray Song
Implementing a Simple Interpreter

16. Off-side rule
def hello():
print ‘world’
indentation sensitive
Ex: ISWIM(1966), occam(1983), Miranda(1985),
Haskell(1990), Python(1991)
Ray Song
Implementing a Simple Interpreter

17. Off-side rule - Cont.
represented as virtual tokens
INDENT
DEDENT
Ray Song
Implementing a Simple Interpreter

18. Off-side rule - Cont.
represented as virtual tokens
INDENT
DEDENT
Ray Song
Implementing a Simple Interpreter

19. Off-side rule - Cont.
represented as virtual tokens
INDENT
DEDENT
Ray Song
Implementing a Simple Interpreter

20. Example
if a > 2:
print 5
print a
IF a > 2 : NEWLINE INDENT PRINT 5 NEWLINE
DEDENT PRINT a NEWLINE
Ray Song
Implementing a Simple Interpreter

21. Grammatical analysis
Cocke–Younger–Kasami algorithm
Earley’s algorithm
LR parser
Recursive-descent parser
Ray Song
Implementing a Simple Interpreter

22. Grammatical analysis
Cocke–Younger–Kasami algorithm
Earley’s algorithm
LR parser
Recursive-descent parser
Ray Song
Implementing a Simple Interpreter

23. Grammatical analysis
Cocke–Younger–Kasami algorithm
Earley’s algorithm
LR parser
Recursive-descent parser
Ray Song
Implementing a Simple Interpreter

24. Grammatical analysis
Cocke–Younger–Kasami algorithm
Earley’s algorithm
LR parser
Recursive-descent parser
Ray Song
Implementing a Simple Interpreter

25. Tools
Bison
Can generate C, C++ and Java codes
ANTLR
Ray Song
Implementing a Simple Interpreter

26. Tools
Bison
Can generate C, C++ and Java codes
ANTLR
Ray Song
Implementing a Simple Interpreter

27. Tools
Bison
Can generate C, C++ and Java codes
ANTLR
Ray Song
Implementing a Simple Interpreter

28. Expression
Precedence climbing method
Shunting-yard algorithm
Parsing expressions in infix notation
Output in Reverse Polish notation (RPN)
Output in Abstract syntax tree (AST)
Operator precedence parser
Ray Song
Implementing a Simple Interpreter

29. Expression
Precedence climbing method
Shunting-yard algorithm
Parsing expressions in infix notation
Output in Reverse Polish notation (RPN)
Output in Abstract syntax tree (AST)
Operator precedence parser
Ray Song
Implementing a Simple Interpreter

30. Expression
Precedence climbing method
Shunting-yard algorithm
Parsing expressions in infix notation
Output in Reverse Polish notation (RPN)
Output in Abstract syntax tree (AST)
Operator precedence parser
Ray Song
Implementing a Simple Interpreter

31. Expression
Precedence climbing method
Shunting-yard algorithm
Parsing expressions in infix notation
Output in Reverse Polish notation (RPN)
Output in Abstract syntax tree (AST)
Operator precedence parser
Ray Song
Implementing a Simple Interpreter

32. Expression
Precedence climbing method
Shunting-yard algorithm
Parsing expressions in infix notation
Output in Reverse Polish notation (RPN)
Output in Abstract syntax tree (AST)
Operator precedence parser
Ray Song
Implementing a Simple Interpreter

33. Expression
Precedence climbing method
Shunting-yard algorithm
Parsing expressions in infix notation
Output in Reverse Polish notation (RPN)
Output in Abstract syntax tree (AST)
Operator precedence parser
Ray Song
Implementing a Simple Interpreter

34. Parser combinator
Straightforward to construct
Readability
Parsec
Ray Song
Implementing a Simple Interpreter

35. Parser combinator
Straightforward to construct
Readability
Parsec
Ray Song
Implementing a Simple Interpreter

36. Parser combinator
Straightforward to construct
Readability
Parsec
Ray Song
Implementing a Simple Interpreter

37. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

38. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

39. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

40. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

41. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

42. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

43. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

44. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

45. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

46. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

47. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

48. Impl
Two top-level nonterminals: STMT and EXPR
STMT: SIMPLE STMT ‘n’ | COMPOUND
SIMPLE STMT: EXPR
SIMPLE STMT: IDENT ‘=’ EXPR
SIMPLE STMT: BREAK
SIMPLE STMT: print EXPR
COMPOUND: if EXPR ’:’ SUITE OPT ELSE
COMPOUND: while EXPR ’:’ SUITE
SUITE: many1(‘n’) INDENT many1(STMT) DEDENT
SUITE: SIMPLE STMT ‘n’
OPT ELSE: ELSE ’:’ SUITE
OPT ELSE: /* empty */
Ray Song
Implementing a Simple Interpreter

49. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

50. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

51. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

52. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

53. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

54. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

55. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

56. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

57. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

58. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

59. Impl - Cont.
EXPR: EXPR ‘==’ TERM
EXPR: EXPR ’ !=’ TERM
EXPR: TERM
TERM: TERM ‘+’ FACTOR
TERM: FACTOR
FACTOR: FACTOR ‘*’ ATOM
FACTOR: ATOM
ATOM: identifier
ATOM: literal integer
ATOM: literal string
ATOM: ‘(’ EXPR ’)’
Ray Song
Implementing a Simple Interpreter

60. Example
if a > 2:
print 5
print a
Ray Song
Implementing a Simple Interpreter

61. Example - Cont.
Figure : Parse
Ray Song
Implementing a Simple Interpreter

62. Interpreting
Abstract syntax tree (AST).
Define semantics for each class of nodes
object(atom): trivial
binary operator BinOP(operator, lhs, rhs, RESULT) :- obj1 =
eval(lhs), obj2 = eval(rhs), calc(op, obj1, obj2, RESULT).
Object & BinOP inherit from Expr
Ray Song
Implementing a Simple Interpreter

63. Interpreting
Abstract syntax tree (AST).
Define semantics for each class of nodes
object(atom): trivial
binary operator BinOP(operator, lhs, rhs, RESULT) :- obj1 =
eval(lhs), obj2 = eval(rhs), calc(op, obj1, obj2, RESULT).
Object & BinOP inherit from Expr
Ray Song
Implementing a Simple Interpreter

64. Interpreting
Abstract syntax tree (AST).
Define semantics for each class of nodes
object(atom): trivial
binary operator BinOP(operator, lhs, rhs, RESULT) :- obj1 =
eval(lhs), obj2 = eval(rhs), calc(op, obj1, obj2, RESULT).
Object & BinOP inherit from Expr
Ray Song
Implementing a Simple Interpreter

65. Interpreting
Abstract syntax tree (AST).
Define semantics for each class of nodes
object(atom): trivial
binary operator BinOP(operator, lhs, rhs, RESULT) :- obj1 =
eval(lhs), obj2 = eval(rhs), calc(op, obj1, obj2, RESULT).
Object & BinOP inherit from Expr
Ray Song
Implementing a Simple Interpreter

66. Interpreting
Abstract syntax tree (AST).
Define semantics for each class of nodes
object(atom): trivial
binary operator BinOP(operator, lhs, rhs, RESULT) :- obj1 =
eval(lhs), obj2 = eval(rhs), calc(op, obj1, obj2, RESULT).
Object & BinOP inherit from Expr
Ray Song
Implementing a Simple Interpreter

67. Subclasses of Stmt - Cont.
Assign
eval() need a parameter: Binding (which variable holds which
object)
ExprStmt
Print
Continue (throwing an exception)
Ray Song
Implementing a Simple Interpreter

68. Subclasses of Stmt - Cont.
Assign
eval() need a parameter: Binding (which variable holds which
object)
ExprStmt
Print
Continue (throwing an exception)
Ray Song
Implementing a Simple Interpreter

69. Subclasses of Stmt - Cont.
Assign
eval() need a parameter: Binding (which variable holds which
object)
ExprStmt
Print
Continue (throwing an exception)
Ray Song
Implementing a Simple Interpreter

70. Subclasses of Stmt - Cont.
Assign
eval() need a parameter: Binding (which variable holds which
object)
ExprStmt
Print
Continue (throwing an exception)
Ray Song
Implementing a Simple Interpreter

71. Subclasses of Stmt - Cont.
Assign
eval() need a parameter: Binding (which variable holds which
object)
ExprStmt
Print
Continue (throwing an exception)
Ray Song
Implementing a Simple Interpreter