The document discusses the process of recognizing tokens in a programming language, detailing how to specify tokens and write a program to recognize them. It introduces transition diagrams, which describe states and transitions in a deterministic manner, showing how each token, whitespace, and other inputs are managed. Additionally, it explains how to handle and return token values through a structured approach.

1. Recognition of Tokens
• We now know how to specify the tokens for our
language. But how do we write a program to
recognize them?
if -> if
then -> then
else -> else
relop -> < | <= | = | <> | > | >=
id -> letter ( letter | digit )*
num -> digit ( . digit )? ( E (+|-)? digit )?

2. Token recognition
• We also want to strip whitespace, so we need
definitions
delim -> blank | tab | newline
ws -> delim+

3. Attribute values
Regular Expression Token Attribute value
ws - -
if if -
then then -
else else -
id id ptr to sym table entry
num num ptr to sym table entry
< relop LT
<= relop LE
= relop EQ
<> relop NE
> relop GT
>= relop GE

4. Transition diagrams
• Transition diagrams are also called finite
automata.
• We have a collection of STATES drawn as
nodes in a graph.
• TRANSITIONS between states are represented
by directed edges in the graph.
• Each transition leaving a state s is labeled with
a set of input characters that can occur after
state s.

5. Transition diagrams (Conti…)
• For now, the transitions must be
DETERMINISTIC.
• Each transition diagram has a single START
state and a set of TERMINAL STATES.
• The label OTHER on an edge indicates all
possible inputs not handled by the other
transitions.
• Usually, when we recognize OTHER, we need
to put it back in the source stream since it is
part of the next token. This action is denoted
with a * next to the corresponding state.

6. 0 6
8
7
Start
>
other
=
*

7. 0 6
8
7
Start
>
other
=
For eg: >=
*

8. 0 6
8
7Star
t
>
other
=
For eg: >=
*

9. 0 6
8
7Star
t
>
other
=
For eg: >=
*

10. 0 6
8
7Star
t
>
other
=
For eg: >=
*

11. 0 6
8
7Star
t
>
other
=
For eg: >=
*

12. 0 6
8
7Star
t
>
other
=
For eg: >=
*

13. 0 6
8
7Star
t
>
other
=
For eg: >=
*

14. 0 6
8
7
Start
>
other
=
For eg: abc > pqr
*

15. 0 6
8
7Star
t
>
other
=
For eg: abc > pqr
*

16. 0 6
8
7Star
t
>
other
=
For eg: abc > pqr
*

17. 0 6
8
7Star
t
>
other
=
For eg: abc > pqr
*

18. 0 6
8
7Star
t
>
other
=
For eg: abc > pqr
*

19. 0 6
8
7Star
t
>
othe
r
=
For eg: abc > pqr
*

20. 0 6
8
7Star
t
>
othe
r
=
For eg: abc > pqr
*

21. 0 6
8
7Star
t
>
othe
r
=
For eg: abc > pqr
*

22. 0 6
8
7Star
t
>
othe
r
=
For eg: abc > pqr
*

23. 0 21
6
3
4
5
7
8
start
< =
>
other
return( relop, LE)
return( relop, NE)
return( relop, LT)
return( relop, EQ)
return( relop, GE)
return( relop, GT)
other
=
*
*=
>

24. 0 21
6
3
4
5
7
8
start
< =
>
other
return( relop, LE)
return( relop, NE)
return( relop, LT)
return( relop, EQ)
return( relop, GE)
return( relop, GT)
other
=
*
*=
>

25. 9 10 11
letter
letter or digit
other
*return(gettoken(), install_id())
start
25 26 27
digit
digit
other
*
start