The document details the author's exploration of Parsing Expression Grammar (PEG) and its application in rewriting a programming language parser in Rust. It covers PEG's fundamentals, comparisons with Context-Free Grammar (CFG), and includes examples of statement parsing and abstract syntax tree construction. The presentation concludes with insights on designing languages and the benefits of using PEG for building parsers.

1. 1

2. Outline
 Who am I? Why I did this?
 Introduction to PEG
 Introduction to programming language
 Write a parser in PEG
 No demo QQ
2

3. About Me
 葉闆, Yodalee <lc85301@gmail.com>
 Study EE in college, Microwave in graduate school,
now rookie engineer in Synopsys.
3
Github: yodalee Blogger: http://yodalee.blogspot.tw

4. Why Did I Do This
 “Understanding Computation: From
Simple Machines to Impossible
Programs”
 In the book, it implements a
programming language parser, regular
expression parser with Ruby Treetop,
which is a PEG parser.
 I re-write all the code in Rust, so I did a
little research on PEG.
https://github.com/yodalee/computationbook
-rust
4

5. Introduction to PEG
5

6. Parsing Expression Grammar, PEG
 Bryan Ford, <Parsing Expression Grammars: A Recognition-
Based Syntactic Foundation>, 2004
 A replacement to Chomsky language, by removing the
ambiguity in grammar.
 The ambiguity is useful in modeling natural language, but not
in precise and unambiguous programming language.
6
Language <- Subject Verb Noun
Subject <- He | Lisa …
Verb <- is | has | sees…
Noun <- student | a toy …

7. PEG Basic Rule
 PEG in definition are very similar to CFG, composed
of rules.
 Rule will either:
 Match success: consume input.
 Match fail: not consume input.
 As predicate: only return success or fail, not consume input.
7

8. PEG Basic Rule
 Replace choice ‘|’ with
prioritized choice ‘/’.
 Consider following:
 CFG: A = “a” | “ab”
PEG: A = “a” / “ab”
 PEG: A = a* a
8
Operator
“” String Literal
[] Character Set
. Any Character
(e1 e2 ..) Grouping
e? e+ e* Optional Repetition
&e And predicate
!e Not predicate
e1 e2 Sequence
e1 / e2 Prioritized Choice

9. Some Example
 NUMBER <- [1-9] [0-9]*
 COMMENT <- “//” (!”n” .)* n
 EXPRESSION <- TERM ([+-] TERM)*
TERM <- FACTOR ([*/] FACTOR)*
 STAT_IF <-
“if” COND “then” STATEMENT “else” STATEMENT /
“if” COND “then” STATEMENT
9

10. PEG is not CFG
 PEG is equivalent to Top Down Programming Language
(TDPL)
 Language anbncn is not context-free, however PEG can parse
it with And-predicate.
 In CFG, A <- aAa | a match: odd number “a”
In PEG, A <- aAa / a match: 2n-1 “a”
 It is an open problem that any CFG can be parsed by PEG
10
A <- aAb / ε
B <- bBc / ε
S <- &(A !b) a* B

11. Using PEG
 There are many library that supports PEG:
 Rust: rust-peg, pest, nom-peg …
 C++: PEGTL, Boost …
 Ruby: kpeg, raabro, Treetop …
 Python: pyPEG, parsimonious …
 Haskell: Peggy …
 …
 So why Rust?
11

12. Introduction to
Programming Language
12

13. Simple Language
 3 types of statements: assign, if else, while.
 Support integer arithmetic.
 Support pair, list, function with one argument.
Simple, but actually we can do some complex things, like
recursion, map.
13
factorfun = function factor(x) {
if (x > 1) { x * factor ( x-1 ) } else { 1 }
}
result = factorfun(10); // 3628800
function last(l) {
if (isnothing(snd(l))) {
fst(l)
} else {
last(snd(l))
}
}

14. Abstract Syntax Tree
 Use Rust enum to store a payload inside.
 “Programming” like this:
14
pub enum Node {
Number(i64),
Boolean(bool),
Add(Box<Node>, Box<Node>),
Subtract(Box<Node>, Box<Node>),
LT(Box<Node>, Box<Node>)
…
}
let n = Node::add(Node::number(3), Node::number(4))
Add
3 4
LT
8

15. Abstract Syntax Tree
 All the statement are Node:
15
pub enum Node {
Variable ( String ),
Assign ( String, Box<Node>),
If ( Box<Node>, Box<Node>, Box<Node> ),
While ( Box<Node>, Box<Node> ),
…
}

16. Pair, List and Nothing
 Node::pair(Node::number(3), Node::number(4))
 List [3,4,5] = pair(3, pair(4, pair(5, nothing)))
 Nothing special
16
Pair
3 4
Pair
3 Pair
4 Pair
Nothing5

17. Environment and Machine
 Environment stores a Hashmap<String, Box<Node>>, with
<add> and <get> interface.
 A machine accepts an AST and an environment to evaluate
AST inside the machine.
17
pub struct Environment {
pub vars: HashMap<String, Box<Node>>
}
pub struct Machine {
pub environment: Environment,
expression: Box<Node>
}

18. Evaluate the AST
 Add evaluate function to all AST node using trait.
 The result will be a new Node.
18
fn evaluate(&self, env: &mut Environment) -> Box<Node>;
match *self {
Node::Add(ref l, ref r) => {
Node::number(l.evaluate(env).value() +
r.evaluate(env).value()) }
…
}

19. Evaluate the AST
 How to evaluate While Node ( condition, body )?
 Evaluate condition => evaluate body and self if true.
19
x = 3;
while (x < 9) { x = x * 2; }
Evaluate x = 3
Evaluate while (x < 9) x = x * 2
Evaluate x = x * 2
Evaluate while (x < 9) x = x * 2
Evaluate x = x * 2
Evaluate while (x < 9) x = x * 2

20. Function
 Function is also a type of Node. Upon evaluation, function is
wrapped into Closure with environment at that time.
 Call is evaluated the function with closure’s environment.
20
Node::Func(String, String, Box<Node>)
Node::Closure(Environment, Box<Node>)
fn evaluate(&self, env: &mut Environment) -> Box<Node> {
Node::Fun(ref name, ref arg, ref body) => {
Node::closure(env.clone(), Box::new(self.clone()))
}
}

21. Call a Function
fn evaluate(&self, env: &mut Environment) -> Box<Node> {
Node::Call(ref closure, ref arg) => {
match *closure {
Node::Closure(ref env, ref fun) => {
if let Node::Fun(funname, argname, body) = *fun.clone() {
let mut newenv = env.clone();
newenv.add(&funname, closure.evaluate(env));
newenv.add(&argname, arg.evaluate(env));
body.evaluate(&mut newenv);
} } } } }
21

22. Free Variable
 Evaluate the free variables in a function to prevent copy whole
environment
 Node::Variable
 Node::Assign
 Node::Function
22
function addx(x) { function addy(y) { x + y }}
-> no free variables
function addy(x) { x + y }
-> free variable y

23. Call a Function
if let Node::Fun(funname, argname, body) = *fun.clone() {
let mut newenv = new Environment {};
for var in free_vars(fun) {
newenv.add(var, env.get(var));
}
newenv.add(&funname, closure.evaluate(env));
newenv.add(&argname, arg.evaluate(env));
body.evaluate(&mut newenv);
}
23

24. What is a Language?
 We make some concepts abstract, like a virtual machine.
Design a language is to design the abstraction.
 Function “evaluate” implement the concept, of course we can
implement it as anything. Like return 42 on every evaluation.
24
Concept Simple, virtual
machine
Real Machine
Number 3 Node::number(3) 0b11 in memory
+ Node::add(l, r) add r1 r2
Choice Node::if branch command

25. What is a Language?
 Abstraction will bring some precision issue, like floating point.
We have no way to express concept of <infinite>.
 We can create a language on geometry as below, which
representation for line is best?
 Consider every pros and cons the abstraction will bring.
25
Concept In Programming Language
Point (x: u32, y: u32)
Line
(Point, Point)
(Point, Slope)
(Point, Point, type{vertical, horizontal, angled})
Intersection Calculate intersection

26. Implement a Parser with
PEG
26

27. The Pest Package
 Rust Pest
 https://github.com/pest-parser/pest
 My simple language parser grammar at:
 https://github.com/yodalee/simplelang
 Parsing Flow
27
Grammar Parser
Source
Code
Pest Pair
Structure
Simple AST

28. The Pest Package
28
use pest::Parser;
#[derive(Parser)]
#[grammar = "simple.pest"]
struct SimpleParser;
let pairs = SimpleParser::parse(
Rule::simple, “<source code>")
 A pair represents the parse result
from a rule.
 Pair.as_rule() => the rule
 Pair.as_span() => get match span
 Pair.as_str() => matched text
 Pair.into_inner()=> Sub-rules

29. Grammar <-> Build AST
Number = { [1-9] ~ [0-9]* }
Variable = { [A-Za-z] ~ [A-Za-z0-9]* }
Call = { Variable ~ “(“ ~ Expr ~ “)” }
Factor = { “(“ ~ Expr ~ “)” | Call | Variable | Number }
29
fn build_factor(pair: Pair<Rule>) -> Box<Node> {
match pair.as_rule() {
Rule::number => Node::number(pair.as_str().parse::<i64>().unwrap()),
Rule::variable => Node::variable(pair.as_str()),
Rule::expr => ...,
Rule::call => ...,
}
}

30. Climb the Expression
 Expression can be written as single Rule:
Expr = { Factor ~ (op_binary ~ Factor)* }
 Pest provides a template, just defines:
 Function build factor => create Factor Node
 Function infix rules => create Operator Node
 Operator precedence =>
vector of operator precedence and left/right association
30

31. Challenges
 Error message with syntax error.
 How to deal with optional? Like C for loop
 A more systematic way to deal with large language, like C.
31
compound_statement <- block_list
block_list <- block_list block | ε
block <- declaration_list | statement_list
declaration_list <- declaration_list declaration | ε
statement_list <- statment_list statement | ε
// Wrong PEG
compound_statement <- block*
block <- declaration* ~ statement*
// Correct PEG
compound_statement <- block*
block <- (declaration | statement)+

32. Conclusion
32

33. Conclusion
 PEG is a new, much powerful grammar than CFG. Fast and
convenient to create a small language parser.
 The most important concept in programming language?
Abstraction
 Is there best abstraction? NO. It is engineering.
33

34. Reference
 <Parsing Expression Grammars: A Recognition-Based
Syntactic Foundation>, Bryan Ford
 <Understanding Computation: From Simple Machines to
Impossible Programs>
 <Programming Language Part B> on Coursera, University of
Washington
34

35. Thank You for Listening
35

36. IB502 1430 – 1510
Build Yourself a Nixie Tube Clock
36