Recognition of Tokens
• We now know how to specify the tokens for our
language. But how do we write a program to
recognize...
Token recognition
• We also want to strip whitespace, so we need
definitions
delim -> blank | tab | newline
ws -> delim+
Attribute values
Regular Expression Token Attribute value
ws - -
if if -
then then -
else else -
id id ptr to sym table en...
Transition diagrams
• Transition diagrams are also called finite
automata.
• We have a collection of STATES drawn as
nodes...
Transition diagrams (Conti…)
• For now, the transitions must be
DETERMINISTIC.
• Each transition diagram has a single STAR...
0 6
8
7
Start
>
other
=
*
0 6
8
7
Start
>
other
=
For eg: >=
*
0 6
8
7Star
t
>
other
=
For eg: >=
*
0 6
8
7Star
t
>
other
=
For eg: >=
*
0 6
8
7Star
t
>
other
=
For eg: >=
*
0 6
8
7Star
t
>
other
=
For eg: >=
*
0 6
8
7Star
t
>
other
=
For eg: >=
*
0 6
8
7Star
t
>
other
=
For eg: >=
*
0 6
8
7
Start
>
other
=
For eg: abc > pqr
*
0 6
8
7Star
t
>
other
=
For eg: abc > pqr
*
0 6
8
7Star
t
>
other
=
For eg: abc > pqr
*
0 6
8
7Star
t
>
other
=
For eg: abc > pqr
*
0 6
8
7Star
t
>
other
=
For eg: abc > pqr
*
0 6
8
7Star
t
>
othe
r
=
For eg: abc > pqr
*
0 6
8
7Star
t
>
othe
r
=
For eg: abc > pqr
*
0 6
8
7Star
t
>
othe
r
=
For eg: abc > pqr
*
0 6
8
7Star
t
>
othe
r
=
For eg: abc > pqr
*
0 21
6
3
4
5
7
8
start
< =
>
other
return( relop, LE)
return( relop, NE)
return( relop, LT)
return( relop, EQ)
return( rel...
0 21
6
3
4
5
7
8
start
< =
>
other
return( relop, LE)
return( relop, NE)
return( relop, LT)
return( relop, EQ)
return( rel...
9 10 11
letter
letter or digit
other
*return(gettoken(), install_id())
start
25 26 27
digit
digit
other
*
start
Upcoming SlideShare
Loading in...5
×

Recognition-of-tokens

490

Published on

Relational operator transition diagram, transition diagram of identifiers or digits, Token Recognition, rules to specify and recognize token

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
490
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Recognition-of-tokens

  1. 1. Recognition of Tokens • We now know how to specify the tokens for our language. But how do we write a program to recognize them? if -> if then -> then else -> else relop -> < | <= | = | <> | > | >= id -> letter ( letter | digit )* num -> digit ( . digit )? ( E (+|-)? digit )?
  2. 2. Token recognition • We also want to strip whitespace, so we need definitions delim -> blank | tab | newline ws -> delim+
  3. 3. Attribute values Regular Expression Token Attribute value ws - - if if - then then - else else - id id ptr to sym table entry num num ptr to sym table entry < relop LT <= relop LE = relop EQ <> relop NE > relop GT >= relop GE
  4. 4. Transition diagrams • Transition diagrams are also called finite automata. • We have a collection of STATES drawn as nodes in a graph. • TRANSITIONS between states are represented by directed edges in the graph. • Each transition leaving a state s is labeled with a set of input characters that can occur after state s.
  5. 5. Transition diagrams (Conti…) • For now, the transitions must be DETERMINISTIC. • Each transition diagram has a single START state and a set of TERMINAL STATES. • The label OTHER on an edge indicates all possible inputs not handled by the other transitions. • Usually, when we recognize OTHER, we need to put it back in the source stream since it is part of the next token. This action is denoted with a * next to the corresponding state.
  6. 6. 0 6 8 7 Start > other = *
  7. 7. 0 6 8 7 Start > other = For eg: >= *
  8. 8. 0 6 8 7Star t > other = For eg: >= *
  9. 9. 0 6 8 7Star t > other = For eg: >= *
  10. 10. 0 6 8 7Star t > other = For eg: >= *
  11. 11. 0 6 8 7Star t > other = For eg: >= *
  12. 12. 0 6 8 7Star t > other = For eg: >= *
  13. 13. 0 6 8 7Star t > other = For eg: >= *
  14. 14. 0 6 8 7 Start > other = For eg: abc > pqr *
  15. 15. 0 6 8 7Star t > other = For eg: abc > pqr *
  16. 16. 0 6 8 7Star t > other = For eg: abc > pqr *
  17. 17. 0 6 8 7Star t > other = For eg: abc > pqr *
  18. 18. 0 6 8 7Star t > other = For eg: abc > pqr *
  19. 19. 0 6 8 7Star t > othe r = For eg: abc > pqr *
  20. 20. 0 6 8 7Star t > othe r = For eg: abc > pqr *
  21. 21. 0 6 8 7Star t > othe r = For eg: abc > pqr *
  22. 22. 0 6 8 7Star t > othe r = For eg: abc > pqr *
  23. 23. 0 21 6 3 4 5 7 8 start < = > other return( relop, LE) return( relop, NE) return( relop, LT) return( relop, EQ) return( relop, GE) return( relop, GT) other = * *= >
  24. 24. 0 21 6 3 4 5 7 8 start < = > other return( relop, LE) return( relop, NE) return( relop, LT) return( relop, EQ) return( relop, GE) return( relop, GT) other = * *= >
  25. 25. 9 10 11 letter letter or digit other *return(gettoken(), install_id()) start 25 26 27 digit digit other * start

×