This document contains a lecture on implementing a lexer using a finite state machine. It discusses representing token definitions as regular expressions and constructing a DFA to recognize them. It then shows how to implement the DFA as a state table in Java and provides pseudocode for a lexer algorithm that uses the state machine to tokenize input strings by iterating character by character and transitioning between states. Finally, it outlines a homework assignment to define lexical rules for a language as a DFA and regular expressions.

1. CSE340 - Principles of
Programming Languages
Lecture 04:
Lexer Implementation 1
Javier Gonzalez-Sanchez
javiergs@asu.edu
BYENG M1-38
Office Hours: By appointment

2. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 2
Review
Given the following token definitions (using regular expressions)
t1 = aabb
t2 = aab
t3 = (a | b) *
1. Are the following strings correct?
aaba
a
aab
∑
2. Which are the token for each of them?
4. Which symbols are in the alphabet ?
3. Create a DFA that represents the previous rules.

3. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 3
Review
1. How many words:
-5
-5.5e-5
5-5
2. Which is the difference between these regular expressions?
[0-9]+.[0-9]+
[0-9]+.[0-9]+

4. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 4
Programming a Lexer
Regular
Expresion

5. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 5
Programming a Lexer
Regular
Expresion

6. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 6
Using IF-ELSE
It is not a good idea!
February 13th, 2008 by Rich Sharpe. Posted in Software Quality, Software Quality Metrics

7. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 7
Using a State Machine
1. Put the DFA in a Table
S0 S1 S2 S3
b
0
1
...
Delimiter, operator,
whitespace, quotation
mark
S0
SE
S1
SE
SE
Stop
S1
S2
SE
SE
SE
Stop
S2
SE
S3
S3
SE
Stop
S3
SE
S3
S3
SE
Stop
SE
SE
SE
SE
SE
Stop

8. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 8
Using a State Machine
2. Put the Table in Java
b
0
1
...
Delimiter, operator,
whitespace, quotation
mark
S0
SE
S1
SE
SE
Stop
S1
S2
SE
SE
SE
Stop
S2
SE
S3
S3
SE
Stop
S3
SE
S3
S3
SE
Stop
SE
SE
SE
SE
SE
Stop
// constants
private static final int ZERO = 1;
private static final int ONE = 2;
private static final int B = 0;
private static final int OTHER = 3;
private static final int DELIMITER = 4;
private static final int ERROR = 4;
private static final int STOP = -2;
// table as a 2D array
private static int[][] stateTable = {
{ERROR, 1, ERROR, ERROR, STOP},
{ 2, ERROR, ERROR, ERROR, STOP},
{ERROR, 3, 3, ERROR, STOP},
{ERROR, 3, 3, ERROR, STOP},
{ERROR, ERROR, ERROR, ERROR, STOP}
};

9. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 9
Using a State Machine
STEP 3. Algorithm
void splitLine (line) {
state = S0;
String string ="";
do {
l = line.readNextLetter();
go = calculateNextState(state, l);
if( go != STOP ) {
string = string + l;
state = go;
}
} while (line.hasLetters() && go != STOP);
if (state == S3)
print (“It is a BINARY number”);
else
print (“error”);
if( isDelimiter(currentChar))
print (“Also, there is a DELIMITER”);
else if (isOperator(currentChar) )
print (“Also, there is an OPERATOR”);
// loop
if (line.hasLetters() ))
splitLine( line – string );
}
b
0
1
...
S0
SE
S1
SE
SE
Stop
S1
S2
SE
SE
SE
Stop
S2
SE
S3
S3
SE
Stop
S3
SE
S3
S3
SE
Stop
SE
SE
SE
SE
SE
Stop

10. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 10
Programming Assignment #1
1. Read a File; Split the lines using the
System.lineSeparator
2. For each line read character by character and use the
character as an input for the state machine
3. Concatenate the character, creating the largest
STRING possible. Stop when a delimiter, white space,
operator, or quotation mark and the current state
allowed. If there are more characters in the line, create
a new line with those characters and go to step 2.
4. For each STRING and WORD report its TOKEN or ERROR
as correspond.

11. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 11
Homework
Define the necessary lexical rules for a programming language
Express these rules using a DFA and Regular Expressions
Share them on Blackboard and discuss their correctness with your classmates.
Remember: Using a DETERMINISTIC Finite Automata

12. CSE340 - Principles of Programming Languages
Javier Gonzalez-Sanchez
javiergs@asu.edu
Summer 2015
Disclaimer. These slides can only be used as study material for the class CSE340 at ASU. They cannot be distributed or used for another purpose.