Lexical Analysis (Breaking strings down into important words)

1. Programming Languages
Building a Web Brower
Instructor : Westley Weimer
1

2. 2

3. 3
Lexical Analysis
the process of converting a sequence of characters into a sequence of tokens
Lexical
Analysis
String
(I like you, …)
List of Tokens

4. 4
Token
the smallest unit of a programming language that has a meaning
＂I <b> like＂
word
start of tag
word
end of tag
word
I
<
b
>
like

5. 5
＂I <b>like</b> you＂
I
<
b
>
Like
</
b
>
you
WORD
LANGLE
WORD
RANGLE
WORD
LANGLESLASH
WORD
RANGLE
WORD
word
start of tag
word
end of tag
word
start of closing tag
word
end of closing tag
word

6. 6
Python Lex-Yacc
A computer program that generates lexical analyzers
Lexical
Analysis
String Token
LexInput
Definition section
%%
Rules section
%%
C code section

7. 7
tokens = (
‘LANGLE’, # <
‘LANGLESLASH’, # </
‘RANGLE’, # >
‘RANGLESLASH’, # />
‘EQUAL’, # =
‘STRING’, # “love”
‘WORD’, # like
)
1. Define Name of Token

8. 8
→ t_RANGLES = r’>’
→ def t_RANGLES(token):
r’>’
return token
→ def t_NUMBER(token):
r’[0-9]+’
token.value = int(token.value)
return token
2. Define Type of Token

9. 9
→ htmllexer = lex.lex()
htmllexer.input(htmlcode)
while True:
tok = htmllexer.token()
if not tok: break
print tok
3. Building and Using the Lexer

10. 10
Python Code

11. 11
Output
Input(Htmlcode) = This is <b>my</b> webpage!
→ LexToken(WORD, ‘This’, 1, 0)
LexToken(WORD, ‘is’, 1, 5)
LexToken(LANGLE, ‘<’, 1, 8)
LexToken(WORD, ‘b’, 1, 9)
LexToken(RANGLE, ‘>’, 1, 10)
LexToken(WORD, ‘my’, 1, 11)
LexToken(LANGLESLASH, ‘</’, 1, 13)
LexToken(WORD, ‘b’, 1, 15)
LexToken(RANGLE, ‘>’, 1, 16)
LexToken(WORD, ‘webpage!’, 1, 18)

12. 12
- Rule Order
- White Space
- Tracking Line Number
- Comment
Notice

13. 13
→ def t_STRING(token)
r’”[^”]*”’
return token
→ def t_WORD(token)
r’[^ <>n]+’
return token
Rule Order
?
“Hello”
Ordering token definitions is of prime importance.
First One Wins!

14. 14
→ def t_WHITESPACES(token)
r’ ’
pass
→ t_ignore = ‘ ‘ # ‘ tvr’
White Space
Tokens do not refer to white space

15. 15
→ def t_newline(token)
r’n’
token.lexer.lineno += 1
pass
Tracking Line Number
Lex knows nothing about line numbers

16. 16
→ states = (
(‘htmlcomment’, ‘exclusive’),
)
→ def t_htmlcomment(t)
r’<!--’
t.lexer.begin(‘htmlcomment’)
Comment
Not find words or strings or numbers or tags
→ def t_htmlcomment_end(t)
r’-->’
t.lexer.lineno += t.value.count(‘n’)
t.lexer.begin(‘INITIAL’)
→ def t_htmlcomment_error(t)
t.lexer.skip(1)

17. 17