3. 3
Lexical Analysis
the process of converting a sequence of characters into a sequence of tokens
Lexical
Analysis
String
(I like you, …)
List of Tokens
4. 4
Token
the smallest unit of a programming language that has a meaning
"I <b> like"
word
start of tag
word
end of tag
word
I
<
b
>
like
12. 12
- Rule Order
- White Space
- Tracking Line Number
- Comment
Notice
13. 13
→ def t_STRING(token)
r’”[^”]*”’
return token
→ def t_WORD(token)
r’[^ <>n]+’
return token
Rule Order
?
“Hello”
Ordering token definitions is of prime importance.
First One Wins!